Computational results inherently rely upon the models underlying them exactly as a person’s opinions inherently rely upon and spring forth from their view of the world. In both cases, ad hoc adoption of multiple short-sighted, incomplete, and conflicting models on the basis of current convenience and their support of cherished preconceptions is the enemy of correctly determining sustainable effective behavior. Further, commonly-held models are necessary to communicate proposals, support debate, analyze the eventual results of decisions, and foster ever more effective future action. Science advances by accreting individually reproducible consensus models but this approach fails when approaching the “wicked” problems of social policy which cannot be isolated and individually solved via science’s reproducible reductionist approach. These problems are characterized by circular causalities and contradictory certitudes (as opposed to uncertainty); hamstrung by lack of agreement on such fundamentals as objective definitions of equity, fairness, or justice; and have no “solutions” in the sense of definitive and objective answers. Indeed, it is normally fortunate if even “next steps” can be determined without alienating and enraging some constituency, particularly in the presence of such unsustainable strategies as secrecy, “sound-bite” engineering and outright lying. Previous solutions and systems address subsets of these difficulties with varying success; however, what is truly needed is a single overarching solution. We propose an open access, crowd-sourced collaborative modeling system, dubbed Whately, which includes the ability to accurately model everyone’s reality models (with a strong push towards internal consistency by tracking beliefs across time and issues); tools to identify and highlight differences between models; and an accessible history to empower future actions by documenting decisions and results and assigning the proper positive or negative credit to and for the various individual beliefs involved. Whately should not only assist in improving the current political process but also in creating crowd-sourced “how-to” guides or by being data-mined for common-interest groups, movie/book reviews, and even matchmaking. Improving shared cognition, most importantly by clear communication and documentation will facilitate more trust, better collaboration, a higher collective intelligence, and a better, more sustainable world for all.
“Wicked” Social Problems
As the complexity, interrelatedness, and urgency of global issues increases, our ability to absorb and make sense of the world lags, both as individuals and collectively. Powerful special interests, self-interested politicians, and shortsighted social trends like fundamentalism are ever more predominant in insisting upon simplistic solutions to truly Gordian problems (in contrast to Laurence J. Peter’s insight that “Some problems are so complex that you have to be highly intelligent and well informed just to be undecided about them”) or throwing up their hands saying the only choice is to maintain their preferred status quo.
Further, as Horst Rittel and Melvin Webber point out (Rittel and Webber 1973)
The search for scientific bases for confronting problems of social policy is bound to fail, because of the nature of these problems. They are “wicked” problems, whereas science has developed to deal with “tame” problems. Policy problems cannot be definitively described. Moreover, in a pluralistic society there is nothing like the undisputable public good; there is no objective definition of equity; policies that respond to social problems cannot be meaningfully correct or false; and it makes no sense to talk about “optimal solutions” to social problems unless severe qualifications are imposed first. Even worse, there are no “solutions” in the sense of definitive and objective answers.
The Issue-Based Information System (IBIS) was developed to solve these problems by supporting the “coordination and planning of political decision processes” and guiding “the identification, structuring and settling of issues raised by problem-solving groups” (Kunz and Rittel 1970).
Recognizing all planning and design as a similar process of argumentation (of the designer with himself or with others) led to the use of IBIS as a method of “design rationale” (Nobel and Rittel 1988) and then to Jeff Conklin’s meme (Christensen 2009) that “we are in the midst of a shift from the Age of Science to the Age of Design” where “the problem-solving process is now clearly social” and inventive in that “in place of finding the ‘right answer’, we seek to gain an understanding of possible solutions” that can be brought into existence.
Still, the fundamental problem remains that our tools and methods for collaborative problem-solving, social planning, and governance are still not much advanced from when the America was founded over two centuries ago and society has evolved considerably since then. Events are happening faster, society is becoming more and more complex, and our representative government is under assault by powerful special interests. What we need are systems and processes by which average citizens can become better informed and more deeply engaged in the questions that affect us all. We also need better ways to recognize and divert from mistaken paths earlier and to avoid repeating the same errors again and again.
Argumentation, Rhetoric and Mapping
Argumentation is the social and cooperative enterprise by which we find truth and resolve conflict. The study of argumentation has been around at least since Aristotle’s Rhetoric but we want to move from coercive rhetoric to true collaboration.
Richard Whately (1787-1863) was a Bampton lecturer at Oxford and the Anglican Archbishop of Dublin. He is known for his Elements of Logic (1826), containing the first known instance of argument mapping in the form of generalized illustration, and Elements of Rhetoric (1828) where he suggests that rhetoric is less concerned with investigation and discovery and that the orator approaches the process of rhetorical invention not as an investigator but . . . already armed with a general proposition he will advance and . . . substantive resources, factual and inferred, by which that proposition may be established.”
Stephen Toulmin introduced an argumentation layout in The Uses of Argument (Toulmin 1958), based upon the format of legal arguments but what really caught on was issue mapping, graphical networks that integrate many problems, solutions, and points of view and show the deep structure of a given issue. Issue maps originated as subsystem S-5 of IBIS and evolved from gIBIS (Conklin and YakemBegemanovic 1988, Conklin and Begeman 1989) to QuestMap and Compendium (Conklin et al. 2001) to CoPe_it! (Karacapilidis et al 2009, Karacapilidis and Tzagarakis 2009). Dialogue Mapping™ (Conklin 2005, Christensen 2009) is a “radically inclusive facilitation process” where a trained facilitator (or “technographer”) creates an issue map in IBIS grammar on a shared hypertext display that “captures and connects participants’ comments as a meeting conversation unfolds”.
Argumentation mapping (debate-mapping) is frequently incorrectly conflated with argument mapping, including in Wikipedia, while reality is that it is the quite different building of visualization diagrams for both a) the major philosophical arguments such as “Can Computers Think?” (http://www.macrovu.com/CCTGeneralInfo.html) at the Argumentation Mapping Project at Stanford and b) social problems such as “Mental Health Services Dynamics and Dilemmas” for a Multnomah County, Oregon Task Force (http://stanford.edu/~rhorn/a/kmap/mess/messPortlandmap.pdf) or the “Long Term Care Integration – Problems and Issues Map” for a 20 member task force appointed by the Board of Supervisors of Alameda County, California (http://stanford.edu/~rhorn/a/kmap/mess/messAlamedamap.v7.1.pdf).
Design Rationale and Argumentation Systems
The first Design Rationale systems implemented the Procedural Hierarchy of Issues (McCall 1991) extension of IBIS which added noncontroversial issues and sub-issues. IBIS was then modified to support software engineering by Potts & Burns (Potts and Burns 1988) and extended by Lee (Lee 1991) with his Decision Representation Language (DRL) which defines the primary elements as decision problems, alternatives, goals, claims and groups and focuses more on the representation of decision making and its rationale instead of on design rationale. Based on DRL, RATSpeak (Burge 2005) is the representation language in SEURAT (Software Engineering Using RATionale), taking into account requirements as part of the arguments for alternatives to the decision problems.
Questions Options and Criteria (MacLean et al 1996) is an alternative representation for argumentation-based rationale adding criteria to explicitly describe the methods to evaluate the options, such as the requirements to be satisfied or the properties desired, linking to options positively or negatively as assessments. Other alternative representations include the WinWin which adds negotiation activities, including identifying key stakeholders of the systems and their individual win conditions and negotiation into the front of each cycle of a spiral software development model and the Decision Recommendation and Intent Model (Pena-Mora, Sriram and Logcher 1993) which consists of the intents of each designer, recommendations and justifications. Negotiations are again needed when conflicts exist between the intents of different designers. Accepted recommendations become design decisions, and the rationales of the unaccepted recommendations are also recorded for iterative design and/or system maintenance purposes.
Other published systems and concepts include Belvedere (Suthers et al. 1995), used for constructing and reflecting on diagrams such as evidence maps and concept maps; Hermes (Karacapilidis and Papadias 2001), built on concepts from decision theory, non-monotonic reasoning, constraint satisfaction and truth maintenance to integrate classical decision-making and argumentation principles and a direct predecessor of CoPe_it!; Araucaria (Reed and Rowe 2004), assists in contextual analysis of text by constructing a tree view of the premises and conclusions; ArguMed (Verheij 2003), with a formal argumentation approach; and Reason!Able (van Gelder 2002), provides a well-structured, user-friendly environment for reasoning through the use of an argumentation tree to decompose problems into their logically related parts and identify missing elements identified. Currently available choices include MindDraw (http://info.cwru.edu/minddraw/), educational software providing assistance in the creation and sharing of visual images of ideas and maps of causal relationships and the paired set of Athena Standard and Athena Negotiator (http://www.athenasoft.org), with Standard being designed to support reasoning and argumentation while Negotiator facilitates analysis of decisions and two-party negotiations.
Moving to the Internet and Web 2.0
While the previous systems offer sophisticated design support and have been proven effective in addressing a wide range of concerns in various domains, they do so only in “walled gardens” with limited problems and a limited number of users. Web technologies, such as on-line discussion forums, wikis, and blogs do a good job of encouraging lots of people to express their opinions and share them widely in an explosion of global knowledge sharing through distributed large-scale conversations but the quality of contributions can vary enormously and they are frequently less than successful at supporting collaborative deliberation around complex and controversial questions. Of course, the first stage of General Services Administration and the White House Open Government Initiative’s concept for next generation citizen consultation, an open access software tool and process to elicit expert public participation (https://expertnet.wikispaces.com) is implemented simply as a wiki and a series of forums and seems to have worked out fairly well.
Another approach has been demonstrated by the Climate CoLab (http://climatecolab.org) developed to “harness collective intelligence to address global climate change (Malone and Klein 2007) at the MIT Center for Collective Intelligence (http://cci.mit.edu/). The CoLab combines three key technologies: open modeling, large scale argumentation, and group decision making to produce what they advertise as “simultaneously a kind of Wikipedia for controversial topics, a Sims game for the future of the planet, and an electronic democracy on steroids”. Activity on the Climate CoLab is focused through a series of annual contests. Teams of Climate CoLab members create proposals which are assessed by expert judges, and Climate CoLab members are invited to vote finalists. Debate is structured around an “argument tree” composed of issues, options/positions, pros, and cons in order to self-organize and make it easier for people to see what’s been said, and whether points have been supported or rebutted. Users may comment on all four and vote on options but submissions to the tree are moderated. Proposals are run through the models and take positions on the issues. An in-depth analysis (Gürkan et al. 2010) of an empirical test of the CoLab argumentation platform where a 160-member community created, in 3 weeks, what the team believed to be the largest single online argument map ever built (around 5000 posts) stated that
(i) users were able to quickly and comprehensively explore and map the debate on the selected discussion topic; (ii) substantial moderation was needed to ensure that the argument map was well-organized and users were confident with the argumentation formalism; (iii) considerable out-of-the map communication occurred, possibly as a way to allow for conversational flows inhibited by the argumentation formalism, (iv) formal rating of contributions favored exploration of the map, understanding the debate structure, and improving the quality of content.
Unfortunately, while users seemed to love the forum, they weren’t particularly good at keeping the discussion on track and the argument trees needed continuous pruning and rearranging by a dedicated group of moderators estimated to need to make up 5-10% of the user population.
The best example of the magnitude of what can be done collaboratively is, of course, Wikipedia. After just ten years, Wikipedia has become not only the largest collaborative web-based effort but the largest and most popular general reference work on the Internet. Whately will draw extensively on its example of how certain things can be done, what the effects are likely to be, and what does and does not work.
Wikipedia’s most impressive feature is that this is done entirely by transparent consensus based upon totally open and immediate access with any moderation applied only after the fact (unlike CoLab). While everyone can edit by default and anybody can go in and make any changes they feel like making, social pressures and community norms have proved sufficient to make the project self-policing. While there are some cases where articles are “locked” to prevent vandalism, this is true only of a very small percentage of the corpus. Thus, censorship or imposing “official” points of view is extremely difficult to achieve and quickly fails thereafter. Most often, all notable views quickly become fairly described and a neutral point of view reached. When conflict does arises over neutrality, details are either quickly resolved on the “talk” pages or subjected to a full editorial dispute resolution process (one that allows time for discussion and resolution in depth but can also permits disagreements to last for months).
Since all previous revisions of an article are saved and stored, Wikipedia, by its very nature, resists destructive edits and vandalism and when such occurs, it can generally be reverted in less time than it required. Given the number of dedicated souls that monitor edits to the encyclopedia (and IP tracking, blocking, and auto-reverting to stop bots), very few edits involving misinformation or hoaxes slip through and nearly all of those involve infrequently visited pages.
Like Wikipedia, Whately is designed around the principles of unlimited scope, scale, and access for maximal openness and transparency. Since visualization of argumentation while working collaboratively towards solving a problem can facilitate the overall process (Kirschner, Buckingham Shum & Carr, 2003), Whately is innately tri-modal with interchangeable views of discrete facts and arguments, text, and auto-generated graphical visualizations using concepts from spatial hypertext (Marshall, C., & Shipman, F.M. 1997) like variable node distance and sizing. Further, cognitive load is reduced and bringing neophytes up to speed is simplified via hierarchical data organization with matching “drill-down” visualizations and the automated production of executive summaries and survey papers.
Wikipedia operates according to three core content policies characterized as “neutral point of view”, “verifiability”, and “no original research”. These three policies work harmoniously to jointly determine the type and quality of material that is acceptable in Wikipedia articles. Wikipedia’s goal is to produce articles which 1) represent fairly, proportionately, and as far as possible without bias, all significant views that have been published by reliable sources by presenting each point of view accurately and in context, and not presenting any point of view as “the truth” or “the best view” and 2) demonstrate verifiable accuracy through citing verifiable, authoritative sources. Whately maintains these goals.
Since Wikipedia articles are large, relatively structure-free, not designed for argumentation, and could even use help in cleaning up contradictory facts introduced by minor errors and mistaken “common knowledge”, Whately will operate with much smaller conceptual blocks and have substantial additional functionalities to record and track votes and positions. Wikipedia articles will be able to be analyzed by the Whately extractor and represented as ordered collections of Whately facts and “facts” that conflict with “facts” in other articles will be flagged as “positions” instead and subject to further argumentation via functionality similar to Wikipedia’s discussion pages. For example, when the English Wikipedia passed the 2 million-article mark, different articles said that made it both well over twenty times the size of the world’s largest encyclopedia (the largest edition of the Encyclopedia Britannica) and just larger than the largest encyclopedia ever assembled (the Yongle Encyclopedia). References supporting disputed facts must be supplied and are ranked by user voting.
Similarly, legislative bills and proposed regulations can be broken up with the aid of the Whately extractor into bite-size pieces and politicians eventually coerced by public opinion into clearly marking which points were most telling and the cause of their votes and explaining why alternatives were not viable. Further, tracking the history and documented reasons for changes of opinion will allow voters to differentiate between reasoned changes due to new information and dishonest crowd-pleasing flip-flopping and allows sound-bite abuse to be minimized. Whately-driven polls can settle on neutral wording and a complete set of alternatives before opening voting and the ability to clearly show when an argument depends upon bad primary sources and multiplicative secondary sources will curb many current media abuses.
Canonizer (http://canonizer.com/) advertises itself as “a Wikipedia type collaborative information system with some added structure to handle point of view (POV) information”. Each “topic” has an “agreement statement” that is “much like a Wikipedia article” with a tree of subordinate camps with “position statements” and supporters where support of a lower, more restrictive camp implies support of all higher level position statements including the agreement statement. This POV structure can prioritized, filtered or ‘canonized’ according to the user’s personal preferences by selecting an existing filter (“canonizer”) or by creating their own. For example, a member of a “Democratic” camp may select a canonizer that counts 10 votes for people who are members of that camp and fewer or no votes for “support” from opposing camps. Whately will include similar canonization functionality.
In a solid display of “eating their own dog food”, the home page states that it “is intended that Canonizer LLC be a leaderless organization where many of the decisions are made by a canonization process. How canonizer.com works, who gets paid how much, all the rules, priorities and so forth will all be determined by such a ‘canonization’ process” And, indeed, there is a prominent list of canonized topics that appear to cover the requirements of corporate governance. Everyone can browse canonized information and registration immediately allows you to contribute, vote (join or support a position statement), set persistent preferences, collect a pay check and so on.
Unfortunately, it appears that the tree structure and the lack of maintenance thereof is a serious Achilles heel. If anyone is supporting any camp or sub-camp, all submitted changes go into a review mode for 1 week before going live with all direct supporters being notified. If anyone objects to any proposed changes, which will always happen in large groups, the changes are “black-balled”. Anyone can add a new camp with a more restrictive camp statement beneath any topic or any camp at any time but there is little provision for removing, merging, or re-arranging camps other than tortuously convincing every single active person who might object to move elsewhere and then “recycling” the camp.
In most current argumentation systems, collaborative features and historical tracking functionalities are limited and no or limited attention is paid to data and knowledge management issues or the empirical analysis of actual interactions mediated by argumentation technology with reasonably large user communities. CoPe_it! aims to buck these trends and tackle cognitively-complex collaboration within relatively large but well-defined closed groups by performing information triage via a process of incremental formalization (Shipman and McCall 1994) ranging from simple collection and sharing of data and knowledge items to formal argumentation and reasoning. Like Whately, visualization issues received extra attention with spatial hypertext projections of the workspace as a prominent feature.
At later stages, CoPe_it! moves to an IBIS formalism and provides a structured language for argumentative discourse together with a mechanism for the evaluation of alternatives. Additional reasoning is performed through the expression of preferences, providing participants with a qualitative way to weigh reasons for and against the selection of an alternative, and reasoning and scoring mechanisms from HERMES keep users informed of the most prominent alternatives.
TopicCentral (http://www.topiccentral.com/) is aimed at the “Public Interest Advocacy Community” to address “the increasing difficulty the public and the other stakeholders have in understanding the virtual flood of complex policies” in public policy development and legislation. While the designers’ vision is close to that of Whately’s, they do not comprehend how well and quickly Wikipedia reaches a neutral point of view (with claims like “Truth does not equal NPOV” and “Advocates are constantly at ‘war’ with each other to get their ‘spin’ inserted.”) and their moderator-driven design concept seems to show a total lack of understanding of the magnitude of the problem. After pointing out that “healthcare reform bills are typically between one and two thousand pages each, with extensive cross-referenced modifications to other existing bills, and packed with low-level detail”, they claim that “in contrast, a TopicCentral profile of such a bill is relatively short, easy to read, and focused on high-level, crisply-defined issues” and that for “the knowledgeable sponsor of such a bill, creating a TopicCentral profile will typically require less than a half hour”. TopicCentral also appears vulnerable to bad topic structuring, prejudicial language, and other moderator induced bias.
The 2010 Washington State Ballot Measures Living Voters Guide (http://www.livingvotersguide.org/) is an exemplar for simple, truly crowd-sourced, truly useful debate systems similar to an unmoderated CoLab. With a simple and intuitive user interface, it clearly displays current opinions and reasoning on each of the questions and invites the user to add their own by moving a slider to indicate their current stance, creating a pro/con list either by selecting crowd-sourced points or writing their own, and then updating their slider based upon their reaction to the creation of their list. For each question, the guide divides the slider results into a seven-bar histogram showing the percentage of voters in each and clicking upon the bar shows the key pros and cons for that group. The Living Voters Guide was built upon ConsiderIt, an open source, Ruby on Rails project that is licensed under the AGPL as “an open source deliberation platform that allows people to collaboratively create pro/con lists of the key points around any complex issue.” It is unclear if any data other than that displayed was collected but, in particular, it would have been interesting if data had been collected on the changes of stance on each question as users created their pro/con lists and over time. Whately will collect such data and automatically produce visualizations of the results.
Whately will support the Argument Interchange Format (Rahwan and Reed 2009) and the Legal Knowledge Interchange Format (LKIF), developed by the European ESTRELLA project (http://www.estrellaproject.org/) for legal knowledge representation and the interpretation of law and cases, including their justificatory arguments (Boer, Winkels, and Vitali 2008; Gordon 2008).
Boehm, B. and Kitapci, H. 2006. The WinWin Approach: Using a Requirement Negotiation Tool for Rationale Capture and Use. In Dutoit, A. et al, eds., Rationale Management in Software Engineering: 173-190. Springer Berlin/Heidelberg: Springer.
Boer, A.; Winkels, R.; and Vitali, F. 2008. MetaLex XML and the Legal Knowledge Interchange Format. In Computable Models of the Law: Languages, Dialogues, Games, Ontologies. Lecture Notes in Computer Science, Volume 4884: 21-41. Berlin/Heidelberg, Germany: Springer.
Burge, J. 2005. Software Engineering Using design RATionale. PhD Dissertation, Worchester Polytechnic Institute.
Christensen, K. 2009. Building Shared Understanding of Wicked Problems. Rotman Magazine. Winter 2009: 16-21.
Conklin, J. 2005. Dialogue Mapping: Building Shared Understanding of Wicked Problems. Hoboken, NJ: Wiley & Sons
Conklin, J. and Begeman, M. 1989. gIBIS: A tool for all reasons. Journal of the American Society for Information Science 40(3): 200-213.
Conklin, J., Selvin, A.M., Buckingham Shum, S., and Sierhuis, M. 2001. Facilitated hypertext for collective sense-making: 15 years on from gIBIS. In Proceedings of the 12th ACM Conference on Hypertext and Hypermedia, 123-124, ACM Press.
Conklin, J. and YakemBegemanovic, M. 1988. gIBIS: A hypertext tool for exploratory policy discussion. ACM Transactions on Office Information Systems 6 (4): 303-331.
Gordon, T. 2008. Constructing Legal Arguments with Rules in the Legal Knowledge Interchange Format (LKIF). In Computable Models of the Law: Languages, Dialogues, Games, Ontologies. Lecture Notes in Computer Science, Volume 4884: 162-184. Berlin/Heidelberg, Germany: Springer.
Gürkan, A.; Iandoli, L.; Klein, M. and Zollo, G. 2010. Mediating debate through on-line large-scale argumentation: Evidence from the field. Information Systems 180(19): 3686-3702.
Karacapilidis, N.; Tzagarakis, M.; Karousos, N.; Gkotsis, G.; Kallistros, V.; Christodoulou, S.; Mettouris, C.; and Nousia, D. 2009. Tackling cognitively-complex collaboration with CoPe_it!. International Journal of Web-Based Learning and Teaching Technologies 4(3): 22-38.
Karacapilidis, N., & Tzagarakis, M. 2009. Supporting Argumentative Collaboration in Communities of Practice: The CoPe_it! approach. In Karacapilidis, N., ed., Solutions and Innovations in Web-Based Technologies for Augmented Learning: Improved Platforms, Tools and Applications: 245-257. Hershey, PA: IGI Global.
Karacapilidis, N. and Papadias, D. 2001. Computer supported argumentation and collaborative decision making: The HERMES System. Information Systems 26(4): 259-277.
Kirschner, P., Buckingham Shum, S. and Carr, C. 2003. Visualizing Argumentation: Software Tools for Collaborative and Educational Sense-Making. Springer.
Kunz, W. and Rittel, H. 1970. Issues as Elements of Information Systems. Working paper No. 131, Studiengruppe für Systemforschung, Heidelberg, Germany.
Lee, J. 1991. Extending the Potts and Burns model for recording design rationale. In Proceedings of the 13th International Conference on Software Engineering (ICSE ’13): 114-125. Los Alamitos, CA: IEEE Computer Society Press.
Maclean, A., Young, R., Bellotti, V. and Moran, T. 1996. Questions, Options, and Criteria: Elements of Design Space Analysis. In Moran, T. and Carroll, J., eds. Design Rationale Concepts, Techniques, and Use: 53-106. Lawrence Erlbaum.
Malone, T. and Klein, M. 2007. Harnessing Collective Intelligence to Address Global Climate Change. Innovations 2(3): 15-26, MIT Press.
Marshall, C. and Shipman, F.M. 1997. Spatial Hypertext and the Practice of Information Triage. In Proceedings of the 8th ACM Conference on Hypertext, 124-133, ACM Press.
McCall, R. 1991. PHI: A Conceptual Foundation for Design Hypermedia. Design Studies 12 (1): 30–41.
Noble, D. and Rittel, H. 1988. Issue-Based Information Systems for Design. In Proceedings of the ACADIA `88 Conference, Association for Computer Aided Design in Architecture. University of Michigan.
Pena-Mora, F., Sriram, D. and Logcher, R. 1993. SHARED-DRIMS: SHARED Design Recommendation-Intent Management System. In Proceedings of Enabling Technologies Infrastructure for Collaborative Enterprise: 213-221. IEEE Press.
Potts, C. and Burns, G. 1988. Recording the reasons for design decisions. In Proceedings of the 10th International Conference on Software Engineering (ICSE ‘1988): 418-427.
Rahwan, I. and Reed, C. 2009. The Argument Interchange Format. In Rahwan, I. and Simari, G., eds. Argumentation in Artificial Intelligence: 383-402. Heidelberg, Germany: Springer.
Reed, C. and Rowe, G. 2004. Araucaria: Software for Argument Analysis, Diagramming and Representation. International Journal of AI Tools 14(3-4): 961-980.
Rittel, H. and Webber, M. 1973. Dilemmas in a General Theory of Planning. Policy Sciences 4 (1973): 155-169.
Shipman, F. and McCall, R. 1994. Supporting knowledge-base evolution with incremental formalization. In Proceedings of the CHI ’94 Conference, 285-291.
Suthers, D., Weiner, A., Connelly, J. and Paolucci, M. 1995. Belvedere: Engaging students in critical discussion of science and public policy issues. In Proceedings of the 7th World Conference on Artificial Intelligence in Education, 266-273.
Toulmin, S. 1958. The Uses of Argument. Cambridge: Cambridge University Press.
van Gelder, T.J. 2002. Argument mapping with Reason!Able. The American Philosophical Association Newsletter on Philosophy and Computers, 85-90.
Verheij, B. 2003. Artificial argument assistants for defeasible argumentation. Artificial Intelligence 150(1-2): 291-324.