acm - an acm publication

Articles

Adaptation in large-scale enterprise systems
research & challenges

Ubiquity, Volume 2004 Issue August | BY Kemal A. Delic , Umeshwar Dayal 

|

Full citation in the ACM Digital Library

Deep analytics technologies and the rise of the intelligent enterprise


Observing the evolution of society over the last several centuries, we perceive that the one of the most distinctive human inventions is the business enterprise, the single, common purpose of which is to create value and increase the wealth of society and individuals.

Contemporary business enterprises, embodied in global corporations, are intricate organizational, technological and financial meta-systems operating under dynamic market conditions and uncertain business circumstances. We see them as large-scale, distributed systems characterized by very high complexity. They are typically heterogeneous and very dynamic, involving complex interactions among many humans, applications, services, and devices.

Consequently, enterprises are likely to contain inefficiencies (for example, unnecessary human labor or under-utilized computing resources); they are prone to poor decision-making (due for instance to lack of timely and high quality information, or inadequate modeling and optimization of business operations), and they experience delays and latencies (caused for instance by traffic bottlenecks or imperfect design or engineering). Therefore, it is to be expected that they will change and evolve over time into better forms that exhibit improved performance.

Specifically, in this article we view business enterprises as large-scale, geographically distributed systems with specific dynamics and exhibiting typical behaviors — adaptation being of particular interest to us here, as it seems to be key to the survival of the enterprise.

Adaptation in Natural and Artificial Systems

We observe that survival is a characteristic objective of natural systems, and it might be also an analogous objective for artificial systems such as the business enterprise. To achieve business survival, enterprises must adapt. There is strong anecdotal evidence from successful, long-lasting companies to support this analogy.

Some companies have stayed in the same business for centuries, but have survived by riding the successive waves of technology; an example is Monte Paschi di Siena, a bank from Tuscany, which is still operating as a bank after five centuries. Others have survived by changing their business domain dramatically; an example is Nokia, which was founded more than 130 years ago and has changed its business domain from the pulp industry to mobile communications.

Thus, we postulate that the longevity of an enterprise stems from its ability to evolve and adapt over a prolonged period. One important distinction here is that natural systems evolve under nature-driven selection processes, whereas evolution of the artificial system is driven by intentional, conscious and intelligent decisions, dictated by competitive market conditions.

Adaptive Enterprise Systems

Decision-making within the enterprise covers the gamut from strategic decisions to tactical/operational decisions. Strategic decisions, such as technology acquisitions, mergers and splits, are relatively infrequent, but of high impact to the enterprise, and are made by business executives. Operating decisions are relatively frequent and of low-to-moderate impact. These are strong candidates for automation/mechanization and future improvement. Tens and hundreds of thousands of such decisions are made on each business day within large-scale enterprises. Therefore, we believe that sound, orchestrated decision-making at all levels, grounded in timely, accurate information, across the entire business enterprise and sustained over a long period of time, is essential to the performance of the enterprise and its ability to adapt and survive.

To better understand adaptation in large-scale enterprises, we should devise appropriate models. Given the complexity of large-scale enterprises, and the stochastic and highly dynamic nature of their operations, creating a simple, but useful, model for the large-scale enterprise is one of several hard challenges here. It is important to devise the simplest possible model, while still capturing the behavior of interest (adaptation).

Figure 1. Closed Loop Adaptation

The enterprise can be split into three principal components (Figure 1) forming Wiener's cybernetic closed-loop. The business operations of the enterprise run on top of IT/IS fabrics, from which relevant data and information are provided via the monitoring and management systems to enable the enterprise to act and adapt its operations. All necessary data and information sets are gathered and harvested (1); then transformed into telling performance indicators and operational parameters (2); and finally mapped against business objectives and operational constraints, leading to the execution of the appropriate optimal actions (3). All this is done to satisfy client and customer business requests, while cooperating with partners and suppliers, and conforming to business policies and constraints. This highly idealized model still has certain explanatory value; namely, it is clear that the key to adaptation lies in the monitoring and management systems (where decisions are made).

As a refinement of this simple closed-loop model, the enterprise could be modeled also as a hierarchical network interconnecting several autonomic (self-recovering, self-healing, self-tuning, self-reconfiguring) units. Strategic decisions are typically made at the top of the hierarchy and communicated in time to the distributed, autonomic units, which are then responsible for adjusting their behavior accordingly and tuning their performance in their respective operating environments.

The process of adaptation could be split into re-architecting steps (such as structural changes in the enterprise) combined with fine-tuning (such as incremental, parametric improvements). The challenge is to understand how people, technology and processes can be effectively combined into a large-scale enterprise that continually adapts, not only to changes in the external environment, but also to internally triggered and executed changes.

Enterprise Technologies Enabling Adaptation

The development of various technologies during the last several centuries has enabled business enterprises to adapt, either by improving their performance by orders of magnitude or by enabling them to enter newly created markets efficiently. Some recent investigations into factors driving productivity hint that technology advances have contributed to more than half of the overall productivity growth in the last 50 years (BusinessWeek, May 17, 2004, p. 81).

Consequently, we might assume that the recent developments in the IT industry have had a positive impact on business performance. Enterprise IT infrastructure plays an important role in enterprise evolution and adaptation. As the IT infrastructure follows enterprise adaptation, it evolves as well: typically through step-wide re-architecting of changes followed by fine-tuning and technology adaptation. This could be likened to house remodeling, which starts with a visionary sketch (dream home), followed by engineering design and blueprinting as guidance for the final remodeling work. During execution, it is normal to interject constant re-adjustments and tuning (guided by functionality, appearance, performance or cost, for instance).

During the last 50 years enterprise IT systems have evolved from the central, glass-house building hosting a mainframe computer, through the era of distributed, omnipresent computers, into modern, utility-based IT fabrics, which may be kept in-house or outsourced. This is an evolutionary, architectural and structural change, which illustrates long-term IT infrastructure adaptation to the evolving business needs. One may observe that enterprise IT has evolved from being a cost center to becoming a crucial, dependable enterprise service.

Yet another way to understand this evolution would be to analyze the IT budget structure and its evolution. Most large-scale business enterprises today are spending around 3 percent of their revenues on IT operations. Large corporations are typically reducing the number of data centers, servers and enterprise applications, but the structure of the IT budget has drastically changed. Previously, 80 percent of budget was spent on keeping IT going and only 20 percent was dedicated to innovation. Now, it seems that the objective is exactly the reverse — to spend 4 in 5 dollars on IT renewal. But, this cannot be done overnight. This is typically done as a 5-year evolutionary process in which the budget is kept constant (resulting from cost reduction and revenue growth), but more importantly, is totally restructured. This important budgetary shift allows greater innovation, which then stimulates development of the new enterprise technologies. Ultimately, this should facilitate better adaptation and faster response to changing needs of the enterprise.

Concomitant with the development of enterprise IT infrastructure technologies the last decade or two have been marked by important advances in technologies for monitoring and managing the infrastructure. The emphasis has been on instrumentation, large-scale data gathering, processing data into information, and calculation and presentation of various statistical performance indicators.

It seems now that the next decade and beyond might be marked by advanced processing systems rendering analyses, simulations, forecasts, predictions, insights, and actionable plans for optimization. All enterprise operations are instrumented and monitored thus creating a huge amount of information captured in logs, traces, messages, transactions, bills, and reports. Rudimentary reporting and visualization technologies provide useful, but limited, information and understanding of the enterprise's operations. Deeper analytics (forecasting, prediction, simulation, optimization) provide insight and lead to the creation of actionable plans. To provide enterprise wide analytics will require both huge IT resources, such as via enterprise grid-like structures, and advances in scalable, incremental analytic techniques that work within the time constraints needed for timely decision making.

Enterprise Management Analytics

We postulate here that adaptation through the widespread use of deep analytics technologies will see the rise of a new type of the enterprise, which we call the intelligent enterprise.

The intelligent enterprise http://www.acm.org/ubiquity/volume_3/issue_45.html learns from its own internal operations, external interactions with customers and partners, and market conditions to improve its behavior and performance. By doing so, it enables better decision-making across the entire enterprise to deal with the previously mentioned inefficiencies. It is through the use of enterprise analytics technologies that the enterprise constantly adapts and improves itself. We call the components within the intelligent enterprise that enable better decision-making Enterprise Management Analytics — to distinguish them from the classical enterprise monitoring and management.


Figure 2. Enterprise Management Analytics

An architectural sketch of the improved enterprise containing analytics shows multiple architectural layers that are engineered appropriately. The bottom layer indicates instrumentation of all the interconnected network devices, servers, applications, clients and access devices in the enterprise IT fabrics, which are the source of millions of events per business day. They are filtered, aggregated and transformed into trouble-tickets, reports and summaries captured in the next, integration, layer. The analytic layer is fed from the integration repositories, rendering views and managing interactions with a wide variety of users. Analytic artifacts are used to create the virtual environments that we call the "business cockpit" and "IT cockpit" — which facilitate very precise, fail-safe control and dependable decision-making. This will enable C-level executives to manage better their daily business activities (e.g., revenue and margin prediction, risk assessment, managing to service level agreements) and to use optimally the enterprise IT infrastructure (e.g., optimizing return on IT investment, capacity management).

Another important dichotomy here is the vertical split into "Intelligent Business Management" and "Intelligent Resource Management", where the two half-spheres of the C-level control should (ideally) communicate, collaborate and coordinate so that the IT fabric serves the business needs, and so that the relevant business events are passed in a timely manner to the IT control system (IT cockpit).

This conceptual, yet idealized picture of the future enterprise still poses several grand challenges. New, more efficient techniques should be devised for the rapid gathering of terabytes of enterprise operational data. They are mainly unstructured and creation of the high-quality data and information repositories which should feed a new class of scalable, dependable and efficient analytic algorithms is sorely needed. We also feel that the entirely new science and art of the interactive analytics should explore business-critical decision-making. Humans are the most adaptive part of the enterprise bringing in unique intelligence but also representing the main source of uncertainty.

Conclusion

To summarize, we believe that the renewed interest in the topic of adaptation in large-scale business systems might lead to a better understanding of this intricate field. This may, in turn, not only enrich the science of complex systems http://www.cordis.lu/ist/fet/7fp-ws3.htm, but also address and resolve some very hard, practical problems facing business enterprises today. This should also clarify the art of creating, managing and evolving these large-scale enterprise systems. As new technologies for enterprise management analytics are invented and deployed, we envision the eventual appearance of a new breed of intelligent enterprise that is able to survive by adapting to changing business conditions.

About the Author Kemal A. Delic [kemal_delic@hp.com] is HP Master enterprise architect and Program Manager with Hewlett-Packard's Managed Services, Global Business Unit with relevant experience in knowledge management, conceptual modeling, and real-time intelligent systems.

Umeshwar Dayal [umeshwar_dayal@hp.com] is HP Fellow and Director of the Intelligent Enterprise Technologies Laboratory at Hewlett-Packard Laboratories. His research interests are data mining, business process management, distributed information management, and decision support technologies.

COMMENTS

POST A COMMENT
Leave this field empty