If we want to advance the art and science of software development, we should direct our attention to the insect world of stigmergy.

Why Can’t Programmers Be More Like Ants? Or a Lesson in Stigmergy

In the good old days before the Internet vanquished time and distance, software developer teams were autocratic organizations as shown in Figure 1a. Face-to-face teams developed in a top-down linear fashion under the auspices of a chief programmer. This required central planning and control, and strict adherence to sequential “waterfall” methods. The time and effort to develop a program of m lines of code grew exponentially, mt, largely because of this organizational structure. The resulting program structure mirrored the top-down structure of the organization according to “Conway’s Law”—that the structure of the system mirrors the structure of the organization that designed it [1 , 2]. For the most part, software systems mimicked the structure in Figure 1a.

 

Figure 1. (a) Before the Internet, program structure mirrored organizational structure as a top-down autocratic team. (b) After the Internet, program structure mirrors distributed network-like organizational structure.
Figure 1. (a) Before the Internet, program structure mirrored organizational structure as a top-down autocratic team. (b) After the Internet, program structure mirrors distributed network-like organizational structure.

But then something unusual happened. Software structures underwent a radical change as their development teams became distributed across time and space, as shown in Figure 1b. Instead of a centrally planned, top-down, linear structure, code structure became network-like. Shortage of programmers, lower labor costs in emerging countries, and lifestyle preferences of the extreme programmer all contributed to this tectonic shift. As a consequence of socio-economic pressures and the Internet, a different kind of development process emerged. And in keeping with Conway’s Law, the structure of code followed suit.

The code base has become the central actor in both organizational and program structure. Instead of a top-down, hierachical structure, code now mirrors an organizational structure whereby the code base itself is central, and programmers (actors) are distributed. But, the distributed team of actors made additions, deletions, and modifications based on a set of distributed rules instead of a centrally planned schedule of behaviors.

The resulting distributed network-like organizational structure and its corresponding program structure have more in common with ants than humans. Like ants, bees, and termites, software development has become stigmergic [3]. The emergence of stigmergy among programming teams has gone largely unnoticed. But once we get it, software development teams can improve quality and lower costs even more by duplicating ants, bees, and other insects that have perfected stigmery over the past million years. Here is where stigmergy fits in the new organizational structure of Figure 1b, which dictates the organization of both code and people.

Stigmergy

The French entomologist Pierre-Paul Grassé described a mechanism of insect coordination he called “stigmergy”—work performed by an actor stimulates subsequent work by the same or other actors [4 , 5]. That is, the state of a building, code base, highway, or other physical construction determines what needs to be done next without central planning or autocratic rule. The actors—insects or programmers—know what to do next, based on what has already been done. This intuitive urge to extend the work of others becomes the organizing principle of modern software development.

Stigmergy ensures tasks are executed in the right order, without any need for central planning, control, or direct interaction between the actors performing the work. Stigmergy solves the coordination paradox problem of how insects of very limited intelligence, without apparent communication, manage to collaboratively tackle complex projects, such as building a nest or writing a program. For example, ants find the shortest route to food in optimal time, and simple-minded termites build hives without central control mechanisms or intelligence oversight. Programmers know what to do by looking at the code base and email messages from fellow programmers. A chief programmer is not needed to dictate the next state of the code base

Grassé coined the word stigmergy from the Greek word, stigma, which means “mark or puncture” and ergon, which means “work, action, or the product of work.” According to Gassee, stigmergy is “the stimulation of workers by the very performances they have achieved.” Modern programmers build software by stimulating each other to add, modify, or delete code segments based on the current state of “work, action, or the product of work.” That is, they modify code based on the state of the code base [6]. Tesseract is a tool for facilitating stigmergic code development in a distributed team organized according to Figure 1b [7].

The organizational structure of Figure 1b is completely compatible with stigmergic software development. At its core is the code base—the object to be constructed—and the distributed, loosely managed programmers spread throughout space and time, but coordinated by a combination of stigmergic intellect and markers embedded with the code base—markers may be as simple as email or comments, or as sophisticated as code reviews.

Ants use a special type of chemical marker—pheromones—to highlight the results of their activity. This type of indirect stimulation, not by the work itself but by special markers, is called “marker-based stigmergy.” Similarly, distributed programmer teams involved in cooperative code development across the Internet lay down analogous pheromones for their unseen and virtual collaborators when they augment code development by email interaction, comments, phone calls, documentation, and webcast meetings [8]. In fact, the stronger the marker system, the better the resulting code [9].

The essence of modern software development is stigmergic intellect and markers embedded within the code base. Markers make stigmergy more efficient, by more reliably focusing a programmer’s attention on the most relevant aspects of the work that needs to be done. However, it incurs a cost because programmers must perform the task of manufacturing markers in addition to the work itself. But, in the Internet age of global software development, this cost is more than compensated for by speed, agility, and lower costs.

If we want to advance the art and science of software development, we should direct our attention to the insect world of stigmergy. This means fully comprehending the value of decentralized collaboration and pheromones or markets embedded in the code, itself. If we practice what ants and termites already know, we can build systems that any individual ant, termite, or human cannot fully comprehend. This is the true meaning of collaboration.

 

References

[1] M. Conway, “How Do Committees Invent?” Datamation, vol. 14, no. 4, 1968, pp. 28–31.

[2] J.D. Herbsleb, and R.E. Grinter, “Architectures, Coordination, and Distance: Conway’s Law and Beyond,” IEEE Software, vol. 16, no. 5, 1999, pp. 63–70.

[3] Francesco Bolici, James Howison, and Kevin Crowston, Stigmergic coordination in FLOSS development teams: Integrating explicit and implicit mechanisms. bolici@unicas.it. OrgLab: Department Economics and Law, UniCLAM, Italy.

[4] Grassé, P. P. (1959). La reconstruction du nid et les coordinations interindividuelles chez Bellicositermes natalensis et Cubitermes sp. La théorie de la stigmergie: Essai d’interprétation du comportement des termites constructeurs. Insectes Sociaux, 6(1), 41–80.

[5]  Heylighen, F. Stigmergy as a Universal Coordination Mechanism II: Varieties and Evolution. Evolution, Complexity and Cognition Group, Vrije Universiteit, Brussels.

[6] Dabbish, L., Stuart, C., Tsay, J. and Herbsleb, J. . Social Coding in GitHub: Transparency and Collaboration in an Open Software Repository. In Proceedings, Computer-Supported Cooperative Work, Seattle, WA, 2012, 1277-1286.

[7] Sarma, A., Maccherone, L., Wagstrom, P., and Herbsleb, J.  Tesseract: Interactive Visual Exploration of Socio-Technical Relationships in Software Development. In Proceedings, International Conference on Software Engineering. Vancouver, Canada, May 16-24, 2009,  23-33.

[8]  Kwan, I.,  Cataldo, M., and Damian, D. Conway’s Law Revisited: The Evidence for a Task-Based Perspective.  IEEE Software 29, 1 (2012), 90-93.  doi:10.1109/MS.2012.3

[9] Cataldo, M., and Herbsleb, J. D.  Coordination Breakdowns and Their Impact on Development Productivity and Software Failures. IEEE Transactions on Software Engineering 39, 3 (2013), 343-360.

 

  • Espen Andersen

    I like this – though i think you just described object oriented software development, at least as I remember it from piddling around with Smalltalk many years ago: Extend the code base, occasionally refactor (superclassing) and make sure that the system mimics the real world, to enhance the ability of yourself and others to recognize it and continue developing it.

    I also think that the evolution of tools perhaps played a larger role in this than mere geographical distribution of programmers – the tools made the decentralization possible, but also evolved in response to the need for decentralization to gain access to developers that were geographically remote, for competence or labor arbitrage reasons, as laid out in this old Ubiquity piece, called “Scarce resources in computing”: http://ubiquity.acm.org/article.cfm?id=1386854

    • Ted Lewis

      Is object oriented programming the cause or the effect? Smalltalk and object-oriented programming pre-dates the commercial Internet by many decades, which leads me to believe its more-recent uptake in languages like Java are a response to distributed teams rather than its cause. Ditto for tools.

      But languages and tools are not the central point — the shift from central planning/control to distributed agile control is my point. It is another illustration of Conway’s Law. We simply built tools around this shift.

  • http://www.edpichler.com/ Brad Pickler

    I never thought on that, very interesting. Ants knows what to do without to think. They just continue the other ant work!

  • Tim

    “A chief programmer is not needed to dictate the next state of the code base”

    And yet, projects with a BDFL seem markedly more successful than those without. How do you reconcile this with your claim that a chief is unnecessary?

    There’s a famous orchestra that doesn’t have a conductor, either, yet the vast majority do. A conductor isn’t strictly necessary, then, but having someone in that position does make things go a lot more smoothly, most of the time.

    “If we practice what ants and termites already know, we can build systems that any individual ant, termite, or human cannot fully comprehend.”

    It’s not quite clear, but you seem to be coming down solidly on the “packer” side of the mapper/packer divide. All good programmers that I’ve ever met or worked with are definitely mappers.

    The insect / packer model is good for many things, but I have yet to see it be successful in creating good software. Can you give an example of a good leaderless software project? (I can think of only one.)

    Insects sometimes make things which have the emergent property of being aesthetically pleasing, too, but if you want art, you should hire an artist, not wait outside for your local insect population to perchance create something you like.

    • TGL

      Most open source projects are distributed in the sense that nobody tells contributors what to work on. Yes, a leader decides what to keep in the next revision, but that isn’t a Chief Programmer. It is more in line with stigmergy than hierarchical command.