IEA

https://www.tandfonline.com/doi/full/10.1080/00309230.2020.1739085

(C:\Users\User\Documents\first%20podcast%20sshots\Maxwell%20Robinson%20etc.png) Association for the Evaluation of Educational Achievement

1 Joakim Landahl, Stockholm University Paper to be presented at ECER, Copenhagen 2017 SMALL-SCALE COMMUNITY, LARGE-SCALE ASSESSMENT: IEA AS A TRANSNATIONAL NETWORK The basic idea of the International Association for the Evaluation of Educational Achievement (IEA) was that the educational world could be seen as a natural laboratory. Individual countries were considered as too small and homogenous to provide explanations for differences in school performance. This perspective implied a certain way of organizing research. From its inception as a UNESCO-project in 1958, the IEA was conceived as a transnational collaborative project with researchers from different national and professional backgrounds. This paper aims at exploring some of the ways in which this transnational network was created and how it worked. After the completion of IEA’s six subject survey in the middle of the 1970s, the sociologist Alex Inkeles wrote a review of the nine volumes that had come out of the project. His review was critical regarding the analyses, but he was impressed by the scope of the project and the international community of scholars and practitioners that had made it possible. When I used the word ”prodigious” to describe the IEA enterprise I chose the term advisedly. I cannot think of any other way properly to describe a project which, at least for the science study, administered the same test to strictly comparable probability samples in 19 countries, 4 of them among the less developed, using 14 different languages and reaching into 9,700 schools to contact some 50,000 teachers and 258,000 students. To accomplish something of that kind would seem to require a great reserve of trust and goodwill and a vast network of volunteers, supplemented by combinatorial and entrepreneurial skills of the sort we associate with the multinational corporation, added to still other resources more akin to the discipline in a great international army.1 The work of the IEA is interesting as an example of how a pioneering venture is created collectively and transnationally. International large-scale assessments was something unprecedented, it was a way of working which had to be invented. While the individuals involved in IEA were experts in psychometrics, education, statistics and so on, they lacked expertise in international assessments. Looking back at the early IEA studies, Torsten Husén wrote: “This kind of evaluation was a totally new experience for the kind of educational psychologists like Bloom, Thorndike and myself, who had been specialized in conducting educational measurements in our own countries or in our own cultural spheres.”2 Even the concept of evaluation, launched by Ralph Tyler in the 1930s, was relatively unfamiliar to most of the researchers in Europe, and tests were something that was used for sorting and selecting students, not for assessing the quality of education in total.3 Furthermore, the invention of large-scale assessments, which included organization, knowledge-building, funding etcetera, could only be achieved collectively: in cooperation between a vast number of individuals and research institutes. The novelty of ILSAs was something that had to be learned together. People

1 Alex Inkeles, The international evaluation of educational achievement: a review by Alex Inkeles of International studies in education (9 volumes), by the International Association for the Evaluation of Educational Achievement. National Academy of Education, 1973. p 143. 2 Torsten Husén, Möten, p. 208 [my translation]. 3 Torsten Husén, Jämlikhet genom utbildning (1977) p. 81. 2 who lacked experience of ILSAs were the ones who, together, developed the first ILSAs. By studying their way of cooperation, their network and their community, as well as its tensions, it is possible to highlight the process in which these experts in their respective fields collectively struggled to achieve a new kind of expertise in a field that didn’t exist before. It is a story of cooperation, but also of failed cooperation. It is a story of how a collective invents and develops knowledge, but also how it discovers its limits. The family The IEA was during the early years a small-scale organization that conducted large-scale surveys. The “IEA family”, as it has sometimes been called, consisted of people who knew each other, sometimes well. Professional roles overlapped with personal friendships. Judith Torney-Purta started her career in IEA with the first Civics study in the early 1970s, and for a while she lived in Stockholm where the IEA headquarter was located for some years. She describes the atmosphere as familiar, and indicates that it has subsequently changed: When I began with IEA forty-some years ago it was a loosely coupled network of academics who knew each other and each others’ families. One of my early memories of IEA is in my small apartment’s living room in the Wenner Gren Center in Stockholm in the early 1970s. We were having a singing party, with Kimmo Leimu and me trading Finnish songs and Ingemar Fagerlind helping with Swedish songs. My daughter Kathy was about 5 years old, and she fell asleep on Torsten Husen’s lap. I remember this as part of the family atmosphere of IEA. I think this is less prominent today.4 Parties provided one way of learning to know each other and to socialize. An example is from 1973 when the first results from the six subject study were released, dealing with literature education, reading comprehension and science (Comber & Keeves 1973; Purves 1973; Thorndike 1973). IEA chose to make careful presentations of the data to the media. Ambitious press releases of all the individual studies were written, giving detailed information of the results. An international press conference, held in Stockholm, was organized, in which the three reports were presented. The press conference, which lasted almost a full day – between 10 AM and 16:30 PM – represented a very concrete meeting between science and the media, where the authors of the individual reports had come all the way to Sweden to present the results. Researchers from New Zeeland, Australia, UK and USA had travelled to Stockholm to participate in the event. Present were also the chairman of IEA, Torsten Husén, and his collaborators Benjamin Bloom and Neville Postlethwaite (Hoover institution, vol 362). It was obviously also an opportunity for the involved people to meet and socialize. In a letter addressed to the participant authors, to the IEA council and to Neville Postlethwaite, the international coordinator Roy W Phillips informed that there would be social events arranged the evening after the press conference, as well as the following evening. “Social events are being planned for the evenings of 24 and 25 May. It would help considerably if you could please bring in the limit of duty-free alcohol and cigarettes.”5 When the international network of scholars was assembled in Stockholm, it was thus used not merely to give detailed presentations to the press, but also to bring in cheap alcohol and cigarettes. The organization was dominated by men, although a few women came to occupy prominent positions. When the leading representatives of the organization met it was not unusual that the

4 Judith Torney-Purta, “50 Years of IEA: Some Reflections about its History” C. Papanastasiou; T Plomp, and E. C. Papanastasiou (eds.) IEA 1958-2008. 50 Years of Experiences and Memories. Nicosia: Cultural Center of Kykkos Monastery, 2011): 582. 5 Roy W Phillips, IEA Memorandum, 26 April 1973, IEA archive vol 362. 3 wives joined on the trips. While the men were working they had the opportunity to socialize, thus arguably also strengthening the professional network. For example, when there was a meeting in Honolulu 1968, where Husén was guest professor, attempts were made by Torsten Husén to book a hotel close to the beach, since “the ladies” weren’t occupied with work during the daytime. These kinds of experiences, can be seen as a way to consolidate the network. The fact that the wives knew each other and indirectly formed a part of the professional network of IEA could make the continuity of the network even more important.6 Another aspect about any networks that must be stressed is the great variability regarding how intense and close some personal relationships might be. To speak of a vast group of persons as a “network” might give an image of a homogenous group of people who are equally close to each other. In reality, the “IEA family” included all sorts of relationships: close, distant, sympathetic and antipathetic. To understand how the network could be sustained it is particularly interesting to note the close relationships. The relation between Postlethwaite and Husén is such an example. They worked close together in IEA for many years. Husén was chairman for 17 years, between 1962 and 1978, and Postlehwaite was working as an international coordinator for the project between 1962 and 1972. During the early years he also got accepted as a doctoral student in Stockholm, with Husén as a supervisor, and finalized his thesis in 1967, based on IEA data. In 1978 Postlethwaite became the successor of Husén as the chairman of IEA. But Husén and Postlethwaite had more than a professional relationship. Husén has stated that their relationship had some qualities of a fatherson-relation. Neville was the only child of a marriage where the father had died when Neville was only one month old. The mother who was a teacher, had to take care of him alone. The lack of a father may have affected his development. My wife Ingrid told me many times that I, both in my capacity as 'Doktorvater' and in a wider sense, served as a father substitute. For many years we had long telephone conversations, in certain periods almost daily. I had the impression that he seemed eager to keep me posted about what he was doing and when so informing me, he implicitly wanted my advice.7 One crucial function of a professional community is to create a sense that what one is doing is important and worth the effort. Postlethwaite has been described as a person with an almost manic work ethic: “Neville stands out to me as a man of perpetual action with a capacity for work that borders on narcomania and who expects something similar from those in his close working environment”, as Husén remarked.8 In a text Postlethwaite reflected on how dedicated his colleagues in IEA had been, and his impressions were mixed. In the early phase, during the mathematics survey in the sixties, the twelve participating research institutes, were enthusiastic over the pioneering project. “Their commitment was total.” Later, and as more institutes from other countries joined, there was according to Postlethwaite less enthusiasm. “One wonders […] if some join only in terms of what they will get out of the study.” At the same time, he had to concede that the organization also had much commitment, and that many donated their services absolutely free.

6 Letter from Torsten Husén to Neville Postlethwaite and Richard Wolf, 1 March 1968, IEA Archive, vol 357. Lägg till Staffan Wennerholm m fl om domesticity! 7 Torsten Husén, ”Thomas Neville Postlethwaite: A Doctorfather's Subjective Portrait”, in Wilfried Bos & Rainer Lehmann (ed.) Reflections on Educational Achievement. Papers in Honour of T. Neville Postlethwaite [XXX] p. 1. 8 Torsten Husén, ”Thomas Neville Postlethwaite: A Doctorfather's Subjective Portrait”, in Wilfried Bos & Rainer Lehmann (ed.) Reflections on Educational Achievement. Papers in Honour of T. Neville Postlethwaite [XXX] p. 2. 4 “Indeed, if the work of IEA had to be fully paid, the budget would be so enormous that IEA would not exist.”9 The machinery The institutionalization of educational research: the rise of research institutes IEA as an international organization presupposed the existence of national research on education. IEA described itself as an international, non-profit-making scientific association. As such, the criteria for membership was clear. Only institutions that carried out research in education were eligible for membership.10 Thus, it was not countries that were members, but research centers that represented the country. Such an international organization would have been hard or even impossible only some decades before, since there were no or few national centers. Pioneering steps towards establishing a national research organization for education were taken in the late 1920s, when the Scottish Council for Research in Education was formed in 1928, followed by similar organizations in Australia and New Zealand and then in the 1940s when a National Foundation for Educational Research was formed in England and Wales.11 In Sweden SPPI was formed in 1944.12 In other countries this happened later, and that establishment was a prerequisite for membership in the organization. In the six-subject survey published in the 1970s national centers from some 20 countries participated. The participating countries were Australia, Belgium (Flemish-speaking), Belgium (French-speaking), Chile, England and Wales, the Federal Republic of Germany, Finland, France, Hungary, India, Iran, Ireland, Israel, Italy, Japan, the Netherlands, New Zealand, Romania, Scotland, Sweden, Thailand and the United States. IEA was, in other words, not an example of Europeanization of educational research, it extended well beyond the confines of Europe. This created a research space which was, arguably, less homogenous and more open to tensions regarding issues such as translation and comparability. IEA was in a sense standing on the shoulders of all these institutes around the world. In January 1969, IEAs fulltime staff was limited to only three persons, while the national institutes that they were cooperating were considerably bigger (about 30 persons per institute was a common figure). The capability of the respective research centers was a crucial factor. In an early brochure, reporting about a meeting in Hamburg in 1960, the IEA stressed that the success of the project depended on “well-equipped national centers of educational research”, and argued optimistically that the postwar decades had witnessed a very positive trend in that respect. There was a “striking general development” since 1945 that included two directions: teachers and administrators were more interested in objective investigations of educational problems, and there was “a marked and accelerating increase in both the number of full time educational research institutions and the

9 Neville Postlethwaite, “Organising Cross National Research Projects” in Postlethwaite (Ed.) International Educational Research: Papers in Honor of Torsten Husén (Oxford: Pergamon Press, 1986) p. 132. 10 Information on the International Association for the Evaluation of Educational Achievement (IEA) 1968, p. 3. IEA Archive, vol 398. 11 Martin Lawn (2004). “The Institute as Network: the Scottish Council for Research in Education as a Local and International Phenomena in the 1930s.” Paedagigica historica, 40 (5 /6), pp. 720–732, ref on p 726. 12 Christian Lundahl, Viljan att veta vad andra vet: kunskapsbedömning i tidigmodern, modern och senmodern skola (Stockholm: Arbetslivsinstitutet, 2006). 5 amount of public money invested in them.” The specific research centers associated with IEA were said to enjoy a “high and growing reputation” in their respective countries.13 At the same time, sources reveal that the national research centers also were the potentially week links of the whole project. Postlethwaite complained, for example, about the situation in Iran where he doubted the accuracy of its school statistics. There were also worries about the continuity of the project in the country as the persons involved in IEA research suddenly left the research institute they had been involved in and started another. This resulted in a major conflict in Iran between two research institutes regarding who was the most competent representative of the country.14 Another aspect is that the national centers resulted in problematizations of what can be compared and how it can be compared. The national centers were normally involved in the construction of the tests, and could both suggest test items and evaluate suggested test items. In some cases there were criticism against test items for not being applicable in some national contexts. The most radical thing that a country could do was to drop off a certain survey. In 1968 Thailand dropped off the planned Civics survey, referring to the “apparent irrelevance of the test items”.15 Hungary did the same.16 Other times the national research centers rather acted as reminders that it is complicated to do comparative research and that issues of translation and local curriculums have to be taken into account. Another interesting aspect of the relation between the international and the national has to do with visions and reality. There was sometimes a sense that people working on an international level had insufficient understanding of how a study could be conducted on a national level. As a person involved in IEA expressed it: The International Committee that is formed to plan a study is sometimes afflicted with a sense of unreality. The committee consists of people who are specialists in a subject area and who dream up all sorts of exotic questions for a study. Their work is often intellectually appealing and sometimes even compelling. However, they are an ivory tower group who have little sense of what is needed to get the information they seek. Their specifications for data gathering can be so exotic that there is no way they can obtain the information they desire. This often results in a tension between the International Steering Committee and the NRC’s [National Research Centers]. The International Steering Committee sets out their requirements in such a way that they appear to be carved in stone and carried down the mountain!17 Living up to the standards (by lowering your own standards) In 1987 the chairman of the IEA, Alan Purves, described what it was like to be a chairman of the organization: “at the moment I could only compare it to being a parent of a brood of three-year olds each of whom has a cut or a broken toy that needs immediate attention.”18 Exactly what he meant by this is a bit unclear, but in another text he was more concrete.

13 International Project on the Evaluation of Educational Achievement (I.E.A.), Bulletin No. 1, Report of the Meeting at Hamburg, 17-22 October 1960, p 5. 14 Letter from Neville Postlethwaite to The IEA Bureau, 21st October 1969. IEA Archive vol 357. 15 Letter from Lamaimas Saradatta to Neville Postlethwaite, January 18, 1968, IEA Archive, vol 59. 16 Letter from Árpád Kiss to Neville Postlethwaite, without date 1969, IEA Archive vol 59. 17 Researcher quoted in Postlethwaite, IEA Memorandum, January 3 1984, IEA vol 157. 18 Alan C Purves, Chairman’s remarks, September 1987, IEA Archive, vol 388. 6 The reports that I have received indicate that the problem of uniform quality is still a problem facing all the studies. One of the hardest aspects of a cooperative study is that of ensuring that all do cooperate, and a major part of the cooperation is pledging to live up to the international decisions concerning sampling, instrumentation, and data preparation and analysis. None of the studies has escaped the problem of having certain centers do things their way. There is no one center that can be singled out as particularly willful or negligent of the cooperative nature of work. […] In an enterprise like that of IEA, each of the participating centers must accept the yoke of cooperation; not to do so jeopardizes the value and even the existence of a study. I spent a day at the International Study Committee meeting of the Pre-primary Study and witnessed the members slowly learning what I meant to cooperate. Each had to forsake their pride and sense that they could do it better. Of course they could, but then they could not have participated in an international study. To some this came as a shock, and they appeared willing to jeopardize the international cooperative nature of the study in order to have it their way. Unable to submit to the yoke, they tended towards disruption.19 Meetings In 1986 the chairman Postlethwaite published a text that described the lessons he had learned from working with the IEA. His experiences stretched back more than two decades. He had begun his career in IEA in 1962 and 1986 was his final year as a chairman. The text therefore reads almost like an advice-text for his successor, and is quite frank in describing different aspects of the organization that didn’t work perfect. This is his description of the social dynamic of an international meeting: Two skills which appear rare are the ability to listen and the ability to motivate committee members to work between meetings. Very often if, for example a Japanese or French colleague is struggling to say something in English, he usually has something important to say. If, however, the chairman is a person having been immersed in quick-firing US research debates, he may not have the patience or sensitivity to listen carefully. This can have negative effects on the morale of the group. Some colleagues have a mentality of working only when they attend meetings. The meetings have to be used for arriving at consensus. When committee members do not work between meetings, this can play havoc with the preplanned timetable.20 The next chairman, Alan Purves, commented on the dominance of English as language of the meetings. He argued that there were many first-rate researchers who had a reading knowledge of English but were less good at speaking. “As a result, many of the meetings are hampered by poor communication, and many of the documents are produced slowly."21 Written communication: letters, letters, letters. However, most of the communication between the participants in an international project such as IEA had to depend on written communication. There was no internet and international phone calls were expensive. Arranging physical meetings was complicated and expensive, and especially when funding was a problem, written communication had to compensate for lack of international meetings. This written communication had its particular problems. Sometimes the writer of a letter

19 Alan Purves, Chairman’s semi-annual report, January 1988, Husén archive 2:189. 20 Neville Postlethwaite, “Organising Cross National Research Projects” in Postlethwaite (Ed.) International Educational Research: Papers in Honor of Torsten Husén (Oxford: Pergamon Press, 1986) p 133. 21 Alan C Purves, “The Evolution of IEA: A Personal Memoir.” Comparative Education Review 1987, Vol. 31, No. 1, pp. 10-28, quotation on p. 25. 7 didn’t get any reply. As someone in IEA expressed it: ”Sending out requests to National Centers (especially for comments on items) reminds one of dropping a stone into a bottomless well.” 22 There was also the problem and uncertainty about how postal communication worked in different parts of the world. Postal communications are uncertain even in the best of times, and strikes in various parts of the world never help. As a result, work between meetings is slow, and when the chairman can never quite tell whether a lack of response means agreement or a postal disruption. Documents, therefore, often have to be redrafted several times. The communication problem necessarily slows the course of an international study, perhaps by as much as a year. With respect to any one of these problems, the only effective reaction is patience, moderated by whip-cracking.23 Communication by mail was not only slow, there was also the risk of misunderstandings. Purves exemplifies with what occurred during the six subject survey in the 1970s, when data processing was done in New York and analyses were done in Sweden. When the first correlations arrived in Stockholm the results were confusing. They showed that interest and achievement in science and literature were negatively correlated. “Four of us spent a day and a half trying to find an adequate interpretation for the result. A telephone call to New York finally produced the reason: the cores on the internet measures were reversed so that a low score meant high interest. Other occasions have not been so humorous or so easily resolvable.”24 What this example also illustrates is a theme that I here cannot elaborate on but which is significant: the role of computers in the early history of large-scale assessments. This is a fascinating chapter in itself. Here it might suffice to notice that while the computers simplified the work, they were also sources of doubts. As Husén expressed it when he with some years of distance looked back at his intense years working on the six subject survey: “We were also caught in a tremendous wealth of information that, due to time limitations, was only schematically analyzed […]. Those who assisted us in planning data processing and statistical analyses were steering us, not we them.”25 Final reflections Two metaphors were recurrently used by insiders to describe IEA as a collective enterprise during its early years. IEA was, on the one hand, a “family”. On the other hand it was, or consisted of, a “machinery”. The use of these metaphors express two aspects of the organization and its ideals. On the one hand there is the small-scale community of friends and on the other hand there is a largescale project that necessarily involves hierarchies, bureaucracy, division of labor etcetera. These two aspects could sometimes come into conflict, but they could also complement each other, since they also share a common trait. What they have in common is that they in different ways have do with the social cohesion of the organization. In durkheimian terms, they might be described as two aspects of solidarity: mechanichal and organic solidarity. IEA probably needed both. As a family it could make people work for free, it was a way of making people reasonably productive although they didn’t have much funding. On the other hand, this didn’t work on everybody. Some complained

22 Researcher quoted in Postlethwaite, IEA Memorandum, January 3 1984, IEA vol 157:108. 23 Alan C Purves, The IEA Literature Study: If I Had To Do It ALL Over Again – I Am. IEA Archive, vol 157. 24 Purves 1987:26. 25 Husén 1979, IEA in retrospect, p 383). 8 that they worked too much for too little money. And a family can’t organize 2 millions of punch cards. (A family has family albums of photo cards, a machinery has punch cards). The creation of a community requires some kind of glue that make it all stick together, and every time period provides both possibilities as well as limits to this. The IEA was starting up in an era before the internet, before cheap international calls, and lack of funding made meetings, especially periodically, expensive. Most of the communication was done by letters. Sometimes they also met physically, and this included work but also different forms of socializing. To meet in person is, according to the perspective developed by Randall Collins, an important way of creating social cohesion. To take part in what he calls interaction rituals can make people “intensely committed to group symbols, such as a new scientific paradigm” resulting in “emotional energy—confidence and initiative—that enables them to do enthusiastic work and attract new members to the group.” 26 This perspective is valuable and relevant, but it should also be stressed that there were clear limits to the solidarity-aspect in IEA. There are several examples of how the cooperative ideal was tainted by problems with communication, cultural differences, language barriers and other kinds of tensions. These problems were to an extent a barrier towards achieving results, but it was also a part of the process. Discovering cultural differences and the limits of comparison was an essential part of realizing what comparative work was actually about. What did IEA achieve? What was its main results? The perspective from this paper makes it possible to answer these questions a bit differently from what one might expect. It might be argued that the main result of the IEA’s work wasn’t the scientific results. Rather, the main results of IEA was the IEA in itself: that it managed, in spite of some of the problems it encountered, to survive. This achievement contrasts with the occasionally very uncertain future of the early IEA. There are numerous examples already from the 1970s, of how the organization discusses whether it had a future or not, and what that future in that case should look like.27 Problems with funding was a major reason for doubting whether IEA would keep on going as an organization. In June 1972 Torsten Husén had a meeting with Champion Ward of the Ford foundation. Ward thought that it would be a pity if the “IEA machinery” would be dismantled after the completion of the six subject survey. He and his colleagues at the foundation felt that the mere fact that the collaborative spirit had been preserved for more than 10 years and that an effective network as well as an impressive amount of competence and experience had been built up, justified the preservation of the machinery.28 This statement indicates that the most fundamental result of the IEA might have been the organization in itself. Rather than the particular results of the study (which today are forgotten) it was the network, the infrastructure, the community that, although it had its problems and limitations, proved long-lasting and influential. IEA gave birth to a number of studies, but it also managed to keep itself alive, and thereby to institutionalize the idea of international assessments. One day (perhaps in the late 80’s) a young man named Andreas

26 Randall Collins & Mauro F. Guillén, “Mutual Halo Effects in Cultural Production: The Case of Modernist Architecture.” Theory and Society (2012) 41:527-556, quotation on p. 529. 27 REF 28 Torsten Husén, IEA Memorandum, 15th June 1972, IEA Archive vol 298. 9 Schleicher, by chance attended a lecture by Neville Postlethwaite. According to Schleicher it was that lecture which made him interested in educational research.29

29 https://www.theguardian.com/education/2013/nov/26/pisa-international-student-tests-oecd