Harvard Graduate School of Education; WGBH Forum Network; Shaping the Field of Educational Research

Good evening everybody. Thank you for being here. I'm Kathleen McCartney dean of the Harvard Graduate School of Education and it's my pleasure to welcome you to tonight's Asquith forum entitled shaping the field of education research insights from the National back stage. I'm delighted that this forum is being held in conjunction with the 13th annual student research conference held here at the school. The theme of this year's conference is the power of Educational Research translating knowledge into action. As Dean it's so gratifying to see the students take the lead in an event like this that showcases our students best work and the work of other students not only across the country but we have international students presenting this year as well. So within this context I'm quite confident that tonight's panelists will have valuable insights about how institutional decisions about education research are made. I want to acknowledge the tri chairs of tonight's Well tonight's event and also the Student Research Conference more generally. They are Jenny Wiener Ana Leah and Meredith Mira. They and their

colleagues deserve a lot of thanks for pulling off this wonderful event. And I also want to acknowledge the faculty tried Cheers Judy singer and Terry Technet. So I think everybody here knows what our mission is here at the Harvard Graduate School of Education. We prepare leaders in education and we generate new knowledge to improve student opportunity achievement and success. And tonight and tomorrow we really turn our attention to the part of the mission that has to do with generating new knowledge building a knowledge base in education is critical for the education sector. It might surprise you to learn that when I meet with. People like superintendents from Chicago and elsewhere they ask me about the knowledge base and education and how they can use it better and how we can produce more usable knowledge to guide their instruction and policy. So tonight I suspect we'll be talking about this is well the panel is planning to grapple with a number of different issues including what topics and methods are considered viable for public consumption.

Funding directives and educational decision making. What are the methodological and theoretical stances underlying these principles and how do they serve to actualize an institution's goal. And I know you will enjoy hearing from tonight's panelists. Russ Whitehurst chemical fire and Michael Pollan and I say just a few words about each of them and then turn the event over to them. Grover Russ Whitehurst was appointed by George W. Bush to a six year term as the first director of the Institute for Education Sciences. As I think everyone in this audience probably knows the institute conducts supports and disseminates research on educational practices that improve academic achievement statistics on the condition of education in the United States and evaluations of the effectiveness of federal and other education programs. Russ has a rich history and an education. He previously served as assistant secretary for the

office of Educational Research and improvement which was the predecessor for the institute and before beginning federal service. He was the professor of psychology in pediatrics as well as the chairman of the Department of Psychology at the State University of New York in Stony Brook. He was vice Academic Vice President of the Merrill Palmer Institute in Detroit and while there he was editor in chief of two leading journals in the field the Merrill Palmer quarterly and developmental review. During his career as a researcher Ross was the author and editor of five books and he's published more than 100 scholarly papers. To his left is Kent McGuire who is the dean of the College of Education at Temple University. He also serves as the director of the Center for Research in human development and education which is a university based research organization focused on the study and demonstration of effective strategies for educating poor and minority children. Kent is a tenured professor in

the educational administration program with the Department of Educational Leadership and policy studies at Temple University. Prior to joining temple Kant was senior vice president of DRC where his responsibilities included leadership of the education children and youth division from 1998 to 2001. Ken served in the Clinton administration as assistant secretary of the US Department of Education where he was the senior officer for the department's Research and Development Agency and he was also an education program officer for the Philadelphia based Pew Charitable Trusts. I think that was 1995 to 1998 where he managed Pew's K through 12 transport folio and in that capacity interacted with our own Bob Schwartz. I'm guessing. And now I want to introduce Michael Pollack our own Michael Pollack as we all know she's an associate professor here. She's an anthropologist of education who studies how youth and adults

struggle daily to discuss and address issues of racial difference discrimination and fairness in schools and community settings. And as we struggle to do this here at the school I'm always so happy to talk to Micah who has it can bring such wisdom to some of the discussions we've been having here in our own campus. Her first book color mute was the winner of the 2005 outstanding book award. Her forthcoming book is entitled because of race how Americans debate harm and opportunity in our schools. And this builds on Mike his experience working in the U.S. Department of Education's Office for Civil Rights where she investigated and addressed claims of discrimination in schools. Mike also has another book about to come out it's called everyday racism getting real hard. Yes. Every every day and you're. Getting real about races. Well that's a little better.

Well but it's probably addressing everyday racism. So there you go. Mike is spearheading a project here the project on the preparation of educators for diversity which is a new national research effort examining efforts to prepare teachers to serve diverse populations. I invite you to all joining me in welcoming the panels and I think right. I don't know why every time they ask me the things they always make me go first you know I'm in Washington I attempt at political humor I guess you don't watch Saturday like live I am pleased to be back I think this is my fourth time at Harvard since. I went to work for the federal government. Either I did OK the previous times or enough time pass that the bad experience washed away. This is an exciting time in education research I think education research is on the front

burner in a way it has been for for a while. It's created opportunities it's created controversies and I think that the issues I was asked to address tonight are certainly very important ones they are essentially what counts for good research and how are decisions made institutionally in that particular particular area. The Institute for Education Sciences is the principal funder of education research in the United States we invest directly about 260 million dollars a year in education research. We have over 500 active grants 21 major national evaluations in the field. And we support education research as well through our. Logic Toodle data collection efforts like the early childhood lodge juvenile study. Many researchers make their careers off of those longitudinal data that are produced by the National Center for Education Statistics. So clearly because we put more money into the field than any other

agency as is appropriate for the federal government in any research research agenda what I fax the nature of the definition of what constitutes good and useful research and affects the direction for the field. So you know that I think you asked the right person to come and address the issue of how some of these decisions are are made. So I asked really sets the market. For education research just as you know the market for health research and NSF sets the market for fundamental basic research and that is as it should be because the federal government by virtue of its constitution has a fundamental responsibility to support innovation and the technical Arts which is the way the founding fathers talked about the research research agenda. The question of. What constitutes good research is

set at different levels. And so I'll try to do it kind of top down and I'll talk about the role that Congress and the customers for research play in that process. And I'll talk about how it plays its role in that process at the level at which I operate and it will drill down to the level at which staff it operates. If you expected a kind of kiss and tell on what really goes on when the phone rings at 3:00 a.m. to people you will not be getting that but I do the best I can to describe to you both the public process and some of the private process that goes into making decisions about the educational research agenda. Top down it's very very important to look at the the statute the law that established the Institute of Education Sciences and replaced a predecessor organization. We are a riot. Both Kent and I had

that last go Education Sciences reform act and while it has many pages and as has various details buried in it at the level of the topic we're discussing here this evening. It essentially said that education research needed to be part of the procedural norms of science. That education research was not apart from the way science works in other fields but rather needed to be fully in keeping with those procedural norms. And what does that mean. It means I think that people who get funded for research by federal agencies and economics or sociology or psychology or public policy. It helped that the people who get funded and are leading researchers in those fields should

be comfortable with the type of research that's done in education and with the people who are getting funded to do that research they should be a common language that is talked across those disciplines and that language consist of methods and approaches. I used to teach a graduate level methods course if you gave me a few hours I could talk to you about about methods and some some in some detail. I'll just point out here that at the broadest level we're talking about an approach to science that consists of hypothesis testing and is undergirded biological disconfirmation. So there are individuals who have. Views and and theories and hypotheses about the way the world works. They try to set up test of those hypotheses and the test are such that it's possible to prove those hypotheses wrong. And if the individual individual putting forward the hypothesis does not generate a rigorous test of it. If it's a processes about

anything important one can bet that somebody else will. And so it's a triangulation of evidence that emerges from the process of essentially trying to indicate whether a hypothesis can be falsified that leads to. A generation of increasing knowledge and increasing certainty about about direction. So there was certainly evidence at the time I took office that education research had drifted from the procedural norms of science. That evidence exists in various places. You could look at Arthur Levine's recent report on the education of education for searchers as evidence for that claim. You could look at the reports from the National Research Council and other places that education research in many cases seem to have drifted into an area of kind of postmodern humanistic thought that had its own value but was separated from again the procedural norms of the

empirical and qualitative social and behavioral sciences. So one source for determining what is good research is the law under which my my office is administered and clearly the law had the thrust that research we fund should be research that a good economist would think is good research that good psychologists would think is good research that good sociologists would think is good research and so we can sum up that general direction as one of the need to support rigor in research rigor to find selecting given a question to be answered. The best possible method that addresses that particular question. Another source for the kind of top down analysis decision making about what constitutes good research is that the customers for the national research agenda in education. One of the first things that I did on

taking office was to commission a review of a survey of the major customers my office is supposed to serve. These are individuals from categories such as Chief State School Officers or superintendents of instruction or teachers or policymakers from Congress or legislatures and they were asked questions about their familiarity with educational research and what they thought of the enterprise and almost universally there was not a great deal of respect for the enterprise of education research or much of the SATs that the research itself was of value to those practitioners and policy makers. They said things like education research seems to be done by education researchers for other education researchers. It provides little guidance to me a classroom teacher about what I'm supposed to do to get through the day. A series of comments of that

nature. So I got to the top level we have a demand for rigor and also we have a demand for something else summarizes relevance. And those have been the watchwords of my agency since since 2001 that we need to fund research that is rigorous and we need to fund research that is relevant. I've come to understand that rigor is a lot easier that relevance and certainly putting the two together is a particular particular challenge. So that's a top level. Now I want to drill down to how the Institute of Education Sciences operates in order to create both rigor and relevance. Three Ps summarize it. One is priorities. The law the institute the education sciences Reform Act required that the person who holds my position as director of us put

forward a set of proposed priorities for the research agenda for the nation and those priorities are subject to public comment and were subsequently to be reviewed by and approved by the National Board for Education Sciences an independent board that's nominated by the president and approved by by the Senate and so that process occurred in 2003 and the priorities were established and have guided what we've done ever sent and the Prior to can can be I think summarized in terms of the outcomes we're interested in and the inputs we think are most important with respect to those outcomes the outcomes very simply are improving academic achievement particularly for students who are at risk. And the inputs we're interested in are those that are under the control of the education system things that a superintendent can bring to bear in terms of trying to improve academic achievement or something that a chief state school officer

could try to do to improve the education system or something that Congress or a legislative body could do in order to affect the education system. And we set outside the definition though understanding its importance. Many factors that affect education outcomes but part of the control of the education system. Poverty for example clearly has huge effects on outcomes on education outcomes. But we do not directly support results or research on ways to alleviate poverty thinking that that's a very broad agenda. And once you move into those sorts of ripple effects you don't have enough focus anymore to spend the 260 million dollars we spend each year in a way that's likely to produce outcomes. So the priorities again are academic achievement and conditions under the control of the education system that affect academic achievement. Second second Pete was people. It was a strong sense just from a management perspective that we needed to get the right people on the bus to get this job.

So we placed a lot of emphasis on hiring and reshaping the education departments research agency and that process at the very least has resulted in a lot of change so that when I came on board there were roughly three hundred eighty staff many very good people in fact most very good people but a lot of sort of out of place in a research agency. I've said before in public I felt like I was asked to take over a law firm and I showed up and there were very few lawyers working there. So we've made a tremendous investment and hiring scientists to come staff the agency that's supposed to fund scientists with over 60. Individuals with technical skills usually Ph Ds and doctors who've been hired during the time that I've been there. And finally we focused a lot on process the process by which people can compete for and obtain federal funding for

education research and the process has had two arms that I think are both very important. One is establishing funding announcements that are clear. Some people find them to clear and to pursue prescriptive. But at least one can go to the IRS website and find announcements for what we're interested in funding. And I think not have too much confusion about what those what those interests are. And the second is to establish a peer review process that allows for applicants for funding to be to be able to compete for funding in ways that are as predictable as I think it's possible for things to be predictable in the context of a peer review system. And so we have established standing panels of prominent education researchers to review research and they do so independently. They score. The applications and those applications are all funded

as long as they get good scores. And my role in that and the staff role in that is to sign the slates and to detect very rarely a serious mistake that's been made somewhere in the peer review process that needs that needs rectified. And that peer review process I think has been fundamental in increasing the supply of research when I came aboard there were eighty nine active research grants There are over 500 now and the number of annual applications have grown from about 200 a year to over a thousand a year. And I think that's because it's a process in place that the research community considers lawful and predictable enough to be worth the considerable effort that goes into writing a research grant and submitting it to submitting it for funding. The funding announcements themselves. Are built around what we define as our as our fundamental responsibility giving

our priorities. And that's a den of following things that work to enhance academic achievement. With then. The goal of determining what works. We fund four types of effort. One is in the category of discovery. It can be worked with large databases to explore relationships among variables. It can be qualitative work to examine what's going on in classrooms the idea is to generate hypotheses by extracting information from the field using a variety of methods. Second goal is development. You have an idea you have apostasies something needs to be developed it could be a curriculum it could be a way of providing professional development for teachers we find that you develop something the question is does it work. So we fund. Research that addresses the question of efficacy that is to individuals and students who are exposed to this thing whatever it is. On average do better than

those who are not exposed to it. And finally we funded development of measures and tools that support the whole of the whole enterprise. So that's what we do and the processes I think are open and fairly explicit. So there's not a lot. Behind the scenes information for me to share with you. That's different from behind the scenes information you would get if you ask your Dean how are decisions made here at the Harvard Graduate School of Education. But I'll do what I can to give you what passes for a for a peek behind the curtain. If you want to know how our announcements are formed the primary responsibility falls to Dr. Lynn who's the commissioner of education research and the acting commissioner of Special Education Research. So she and her staff get together and they propose ideas

and they balance them off in the draft funding announcements and they and they bounce those off me and Lynn and I close the door and we argue about things and we generally come to a compromise or sometimes she convinces me she's right and sometimes because I have the superior position I can serve that that I'm right. And so that's a process sort of bubbles up from the bottom. The program officers are working with grantees around the country they know what's coming and what's not. What questions are getting what they think we ought to be funding that's worth funding that isn't funded yet. And they propose that and we deal with it. I have an additional job which is to protect the institution that is I guess and to let it prosper. And that means that in addition to the bubble up process I try to pay very careful attention to those organizations and individuals who have

powerful effects on the funding of education research and the health of the enterprise and so consistent with our priorities and consistent with the integrity of the process. If Congress is interested in having something funded we are interested in funding it. If the administration needs particular sorts of information and we can do it consistent with our priorities we try to do that. If their powerful interest groups like the Council of Chief State School Officers who feel they have needs are being met and we can meet them we try to do it so we operate in a political context and part of the politics the legitimate politics of education. Dealing with public need and that public need is expressed through interest groups as expressed through Congress it's expressed through through the administration. Again consistent with our overall ways of operating we try to we try to serve those interest groups. I will point out to

you that the education research community is not a powerful interest group in education research and it's something I think the education research community needs to pay to pay some attention to. So when issues arise in Congress that have to do with education research. Congress will hear from the Chiefs. It will hear from an organization that represents the regional educational labs and other and commercial entities that play in the education research space. And they may hear from the education research community but not a powerful way. So it's interesting that the producers of the knowledge base tend to be among the most. Of the weakest of advocates when it comes to policies surrounding that. That knowledge base. So where are we. We are in a process that has represented I think a transformation in the way that education research is managed at the federal level. I think there's a lot of work that remains to be done. It is the work of a generation and

it's not the work of three or four years where as you all are acutely aware in the middle of a presidential election and that will result to appoint a new director for my office and that will generate possibilities for change and some of those possibilities I think will be positive positive possibilities. But I do think where we're to a point in this nation where education research has has come of age it's respected. People want more of it. If you create. Work that people can use it generates an appetite for more. And it's very important that schools of education are training the next generation of researchers play their role and play it powerfully in providing a supply to fill those substantial gaps in the knowledge that exist at this point will for the forseeable future that's true in every field it's dramatically so in education with if where we need to get is the ceiling in terms of in terms of the supply of good and relevant

education research we're a few inches off the floor and we've got a ways to go and I will take your best efforts to get us there. Thank you. Let me just stand up in this state. Well I too want to thank you all for the invitation and of trying to see if any of my students are in the room. To govern what I say Russ I'm going to as I as I listen to Ross reflect on the agency that is I can't help think about what I inherited when I got there and I I can only confirm

much of Russell's operation it was clear that we were a research organization. Actually we were more of a program improvement shop with lots of staff. But I am intrigued with this. On the central question about principles guiding decisions about research in education and I have my introduction simply reveals I can't hold a job I've done so many different things that I can think out loud with you about that issue from a number of different institutional perspectives so I certainly learned a lot when I was in town in Washington back in what now feels like the prehistoric times. In terms of the norms associated when I got there associated with soliciting and procuring research

I think it was a time characterized by an absence of clear priority and. And weak and inadequate central mechanisms for establishing. I remember very well that when I got there the money was spent and the centers existed. Priorities board was formed with nothing very much to talk about nothing very much to do. And and I think the other thing that was so striking to me was that nor did we are at the time have the legislative authority really to enforce a set of clear priorities that we we had been through about a 10 to 15 year period where most of the keys had been handed to the field. Now there's an argument

for that. There's an argument for that which says the field knows best what the issues are that deserve a study by. But I think we had gone way to one side and needed balance in thinking about what's in the public interest. What kinds of problems or questions deserve federal investment. What kinds of capacities needed to be organized to go after those questions in a in a more systematic way so I found myself in that life. In the early stages of transition from a world in which anything into one in which at least a limited set of important questions might receive some consideration the question for me was what legislative authority would allow an agency actually do that

well and what kind of federal leadership or national leadership is associated with organizing to do its best thinking about where additional resources needed and capacity needed to be built. But then I was gone. And then I was gone. I left a little bit of money on the table for us. A few planning groups you know you know to start with. But I didn't leave him having to re compete. Everything he had because he didn't come to the agency with enough capacity to do that. He'd have been nervous about it I'm sure. But but it was a real adventure. When I went to him DRC I I was just struck.

I had come out of one organization where anything went into one that knew exactly why it exists. Absolutely no question about what its signature knew what kinds of questions it wanted answers to. Therefore it knew precisely how to attack those questions from a methodological point of view. And the standards governing what constituted rigor quality work were exceedingly clear. So I learned a lot about about from that. And then I showed up at school. It was like night and day. And I have learned a lot since being in one. But we're anything but clear about anything. In a way a wonderful place just as I'm sure this school is. But you know competing conceptions about the world.

Many see it as something to be constructed. There are a few who actually want to put the world back together you know or intervene. And so that's one dynamic that's alive and and well in my in my school. We have competing conceptions of what constitutes rigor and quality. No idea about that. The agreement for the most part in the academy is to disagree. About such things. Or I say I should say it more accurately to respect the disciplinary norms that exist. And you accept the idea that the field of education is an applied enterprise where many disciplines and code can co-exist.

So I suppose this is a useful sequence segue to one or two of the questions that I understood were to frame our brief remarks. I made some initial assumptions about which questions I should answer given the fact that Russell was here. I can mention briefly I can talk about the process by which some topics are considered viable. You know what's hot right now. And how did they come to be so. Russia certainly right. One way of answering the question about what's hot is you know where is the where's the money to study. So the government has an enormous market shaping capacity and I would argue responsibility to try to take responsibility for it. For understanding where the where the big

questions are. So. You know another question had to do was moving toward a favored methodology. If there is one I'll speak about that and in a third had to do with what. What kinds of things drive considerations about research at my institution at Temple. I'll talk very briefly about about that in terms of watched by a ball in What's Hot. I don't think there's a single path to relevance or usefulness. If that's another way of putting the question one pathway. Ross alluded to it and has to do with the extent to which the research is aligned with a real question problem of practice.

And you know I think about the outburst a working adolescent literacy as an example of a place where considerable attention has accrued resources have been invested. Why because we are pretty clear around the country that there are just we have a big problem with kids who never learn to read. By the time they were in the second or third grade and their access to the rest of the curriculum. And as a consequence Basler readers were written for kids who are 13 14 years old and so forth and so on. Field initiated research are initiated is still a legitimate category for for thinking about new new work. You trust the field. Whether it is always interested in a problem of practice or policy trust the field to be

close to the contours of existing or prevailing knowledge and to have good instincts about where opportunities to extend knowledge get it new questions or dig deeper into existing ones. Both test existing theories and can for new ones. Those are all very legitimate motivation for what's considered a viable viable work. Of course so much I think depends on where you're having these conversations and both with respect to what's viable and what's hot. There's a book. On this question Well let me back up. I do think that one thing that's interesting today is the role that evidence seems to be play

in education. I mean I think our whole economy is you know has a performance orientation associated with it but evidence now seems to matter a lot. And I think it's actually true that the Dean mentioned that when she's approached by superintendents from Chicago and L.A. the big cities that they want to know more about what works they want better evidence. I think that's pervasive. And it is spawning a very interesting debate about questions of effects. You know what constitutes big. And and I think there's a real premium on evidence. I was about to mention I've been reading a book by Henry Marx called the progress of experiment. It's actually an old historic history from 1900 to 1990. Science is in pharmacy is a study

of randomized clinical trials in the pharmaceutical. And medical sectors. But it tails I think an intriguing story about a tension between the clinicians and the statisticians. And I would assert that in education these days we're on a comparable trajectory. I think in some measure to Russ's with the Institute but in general because of the evolution of large datasets of technology. Sort of a growing appetite for solutions. What have been just enduring and tractable problems. So it's an interesting read in terms of

watching what's happening in education today. If you ask have other sectors been through some of what we are currently experiencing. My sense is the answer to that question is yes. So what's that debate look like. I mean to overstate it only slightly. I have people in my school who would argue that context is everything. And so they would say you know average your facts are you know irrelevant. Context is everything. I have other people in my school that would say if you look at things long enough if you look at them over time if you have a big enough sample sizes if you look across enough studies average of facts. Math and that that you know the truth lies out there

someplace if we just would we would just find I hope the truth is somewhere between those two extremes and that as we think about legitimate researchable questions in education that what we'll see is an appetite for strong designs that address contacts process and treatment effect that is I think what we need more of. Let me comment briefly on how we make decisions about research at Temple. In the college of education the extent to which we could claim to be actively I'm not sure that's always what we do. And I would say that at one level we respond like everybody else to what funders are interested

in supporting. So we look to yes we look to the National Science Foundation. We look to see age and increasingly in a few other areas where some of the work that we have an interest in and capacity to do a Center for Disease Control Department of Labor. Now and again. And so we do that. Increasingly our attention is drawn to our immediate contacts. I'm actually very pleased about that because five years ago when I came to Temple I was struck by how disconnected the work we seem to be doing was with the problems people were struggling with right in our own back yard. So either through hiring or through

persuasion or nudging or incentives now to do that. We've seen a good deal more attention at the college to questions that are of some real significance and importance in Philadelphia in the in the region. Many people who are on the faculty now are probably here because of what temple what Philadelphia serves us. It's important for me to know. That the university is going through something of a transition itself from that of a teaching organisation to one of a research organisation and so the composition of the faculty is clearly demonstrably different than it was five or six years earlier.

But the dominant norm still is for the individual faculty member to pursue things of his or her peculiar interest and which certainly derives from their specific research agenda. We're not going to alter that in any significant way. There is I think a healthy tension in the building between the needs and interests of the field. On the one hand with those of the Academy on the other any of you who are going to move into the Academy will feel this tension. When my phone rings people are calling with big props. And they're usually framed in broad strokes. They're messy multi dimensional and they might take two or three years and a lot of

Russ's money to actually solve. And that's at odds with the faculty especially the ones on the tenure track. Who need problems that they can break apart. Isolate attack. Relatively quickly so that they can write about them in dude. Do chords and get them appropriately placed in the best possible Journal. So that's a healthy tension when you know the work will have to live in a world where both past has to happen but I think the future of bad schools is figuring out how to mediate that tension. Organize faculty and their particular interests and talents in such a way that they can do consequent consequential work

for which they can be rewarded professionally and at the same time pursue questions that are of service to the field I think that's that's where we're headed. But the faculty composition in my college is also affecting the kinds of questions that we pursue. Six seven eight years ago the vast majority of my colleagues were themselves prepared in at school. Today we have psychologists sociologists economists the faculty and they are bringing with them their disciplinary traditions and their tool boxes with them to a set of educational questions and problems and I think were just are demonstrably more interesting organisation as a

consequence both for the kinds of collaboration we're beginning to see between in the moon the faculty inside the college and the colleges relative ability to engage faculty across the university. Just very rich budding collaboration in places you might never have expected to find it before. For instance you know we don't typically get along with the with the people in science and technology. Because the people in science technology don't seem to care about learning and teaching. And but we actually have a set of new demonstrations up and running 3 in math and science where the college of education is largely about looking for and studying the

effects of these new interventions in preparation. So very interesting times because of the different mix of talent new design new questions. And I think increasingly new in healthy balance between questions of social justice which you would find in our mission statement if I read it to you. And you know a growing appetite and interest in intervene and that I think of very rich and useful and appropriate posture for for a contemporary. It's who we exist at the end of the day I think because there's a profession out there and if we don't position ourselves to address the issues that the profession has you know I think

I think we deserve some of the criticism that it currently comes our way. What would I say to students as aspiring scholars and researchers. Just the three things I think. One I think these are in some sense the best of times because the education questions are still front and center and they need and deserve your your attention. We have technologies we didn't have even 10 years ago at our disposal. New software that allows us to come to our questions with greater precision and in a variety of ways

new tools and the problems are abundant. It's one thing to be in a world where you're busy studying things and nobody knows there's a problem. It's another to be in a world in which everybody knows their big problems and they actually want help. And this is one of those moments. You know I would not want to be if I were Ross I wouldn't want to be running the department of energy right now. You know partly trying it for maybe the car I'd like to see yet another environmental protection agency or some place you just get the sun dust isn't shining on them. And so life's not very exciting but this is I think these are exciting times. And in a time when we need an influx of new talent and

imagination and I couldn't think of a better time to be doing or doing this work. Do I think van that we might want. Emerging federal policy to strike a little more balance between issues of process context and effects. Yes. But do I think we've paid too little attention for too long to questions of facts and impact. Absolutely. So I think what Rush has been up to has been really important in terms of setting the field up for a force for a better balance and you know in a way it's been a high stakes endeavor in which you've been studies that you launched the war.

If they don't the Congress might lose interest in the sector altogether. And and that would be unfortunate. So I will sit down with one minute left. And look forward to whatever change we might we after Michael. Thank you. Thank you. OK you've heard a lot of time to keep my remarks short I was given 10 minutes come in under that and I said You promise in a slightly different voice I'm speaking as a researcher coming up in the field. And in honor of my four year old daughter I thought I would organize my shore marks around the thought experiment which is to vault directly to full control of the field. And if I were the Fairy Queen of educational research what would I accomplish with my magic wand. And the first thing I would do is to get us all together I would stop time as the

escalators went up and down with all of us in our suits and I would ask us an existential question are we doing what we went into this field to do. Are we in fact improving schools with educational research first I would ask Are we asking all the necessary questions I think. Whitehurst it's what you were getting at are our questions relevant are we answering questions that are plaguing communities schools and districts. Are we answering questions that real people serving children think are essential to have answered or answering questions children and parents are struggling with on a daily basis. And second are we doing good educational research that allows us to answer those crucial questions. So in my role as fairy queen with no one no one sort of able to leave my presence I would particularly ask like McGuire why in our quest for good research would we ever prioritize any research method over any other apriori. Does it not depend on the question that needs to be answered. The National Research Council in 2001 the report scientific research and education includes as its

third bullet point of what constitutes such research research that uses methods that permit direct investigation of the question. So as fairy queen I would see these simplistic preferences for particular methods and announce once and for all that no method should be valued over others are priori. But it depends on the question that needs to be answered so a rigorous approach. I believe that the word rigorous should not be used interchangeably with quantitative or a rigorous approach means an approach carefully designed to answer a question. So in my career thus far I've tackled issues of race that were tearing apart educational communities and really distracting from effective service to young people and inquiring into these questions people struggling to talk about race issues people struggling over which acts toward children of color were racist. Inquiring into those questions required ethnographic methods to answer the question why because I had interactions I needed to study in real time. Because I wanted to study and needed to study communities real time experiences over time. And so

I chose methods that were essential to answer essential educational questions I used methods that permit direct investigation of the question. So as fairy queen I would make clear to all of us in a way that valuing certain methods over others our priori doesn't really serve the project of figuring out how most regularly rigorously to answer the questions that demand attention as we figure out how to serve children better. Next I would ask my audience the following question. Are we using our position as researchers to clarify and synthesize knowledge in the field and to make it usable it is something that has come up several times now to the people who need to apply it rather than just producing more of it I think you can see whenever you go to the pressure to simply produce more educational research more claims that education I don't think we need is more clean as I think we need better organized educational research. So as fairy queen I would now set about reorganizing the field. First what I would do I would create a map of the terrain I think we need to know in education research who is working on

it who is answered which pieces of the educational puzzle because we often set forth just do more but we don't know really what anybody else is doing. We need more clarity in the field on how the various questions we work on fit together. I think otherwise we each answer our small piece without addressing the other ones can there be a more complex longitudinal system of causation than the ones the kind we study. In education I don't think so. What gets Jose to graduation what's gets 100 Latino students to graduation is a mixture of their health care their mothers health care their job their parent's jobs their housing testing policies teachers reactions to students students reactions to teachers reactions the everyday acts of countless educators how parents respond to those acts what's on television it's a mix of all of these things. So we in educational research essentially are attempting to study the most complex system of causation imaginable. And I think we would serve ourselves and children by mapping out how our various in grees fit together I think we waste a lot of time in education battling over which one of us has found the causal link in the

chain when really we need to be seeing how our increase fit together how do insights and health care and the brain and community organizing and everyday struggles over race issues in schools how do those fit together so that we can clarify in these very complex systems of causation. Who needs to do what to serve children I think it would also assist. New researchers to figure out which piece of the puzzle to research and also help students at schools of Ed figure out where to plug in to get Latino kids to graduation. This brings me to my second act as Fairy Queen to reorganize the field. I think we need to do more to synthesize questions actually that have already been answered. We need to do more in educational research to synthesize work that has already been done and shared topics I think a lot of times the people doing the synthesis are journalists people who write articles in The New York Times Magazine et cetera who digest our research and presented synthesize it. Here's what's known about culture Here's what's known about families etc. I think we need to do more of the synthesis ourselves so these syntheses are not inaccurate. My own effort on along these lines is a book called every day and the

racism that's coming out this summer where I invited 100 colleagues in the field and in fact all doctoral students at Huxley who wanted to join me three years ago in a synthesizing project what is our research in the field say collectively about which sorts of everyday actions by an educator counteract racial inequality or exacerbate racial inequality. And in the end we had about 65 authors over three years about 30 doctoral students working together on the project of synthesizing finding cross-cutting themes across researchers who had spent careers studying these issues of race and education. Some of them experimentally. Some of them through ethnography some of them through interviews depending on their research question. My job as editor was to synthesize findings across all of these people and also to pinpoint concrete to do's for educators. And I think we need more not to toot my own horn but I think that we need more such synthesizing efforts as opposed to just simply producing more claims about education. Next project this fairy queen is related to this I would enforce that every piece of research include alongside it some usable tool

for the stakeholders involved in the research. That would pull that gold nugget ideas from the research for use by teachers by policy makers by young people. When we talk about making knowledge usable often we say to policymakers Obviously there are many more stakeholders who need to have research be usable that also means not just pulling out Id gold nugget ideas but presenting those ideas in a language and format that people can actually use whether its a handout whether its a diagram whether its an inquiry tool that allows our research to be used. One aspect of that is creating portals which we have here at huggy we have a usable knowledge portal where people can come access our research. But I think another we need to be brainstorming more 21st century ways of turning our research into tools usable by people on the ground whether its offering our findings to them or figuring out and creating great tools for them to investigate their own communities. Almost done I don't know how much time I have. If I were fairy queen I would clarify several extraordinarily murky constructs in our field I think is a

key aspect of getting rigorous. Again I don't think rigorous and quantitative are the same thing I also don't think that empirical and quantitative are the same thing empirical means rigorously grounded in data. That's what imperial means grounded in rigorous analysis of data. But we have a number of murky constructs that we use unthinkingly without clarifying we do research with them and that murkiness I think may be keeping us from setting forth methodically to improve service to children. The working group of doctoral students who started working on everyday anti-racism with me has turned into a working group interested in tackling a very murky construct in the field which is the educator prepared for diversity this is the type of person that everybody says we need but nobody defines in the same way. And in fact we've been looking at in a published report on this research and there are countless definitions of what constitutes an educator prepared for diversity What skills or qualities does she possess but also how would we know if an educator were prepared for diversity how would we even measure her

preparation. And so we think that we might gain as a field by clarifying this very murky construct by asking in fact what the various stakeholders in the field mean when they call for educators more prepared for diversity. So to finish up this does not mean I want to note narrowing the definition to one definition but rather clarifying key definitions across the field across stakeholders so this brings me to my last act as a Faerie Queene of educational research. I thought I would bring variety back to our definition of improving education. I sense that we're in something of a closed loop in the field where we've come to a point where we can only think about improving education in terms of raising test scores. And that's your only way we know how to measure it and that's the only way we know how to study it and why it is while it is in fact essential to measure educational improvement by measuring whether students perform better on basic skills tests measuring the improvement of education soley by measuring increases in test scores I think really

greatly reduces our ability to think about what improving education actually entails. And as we know it also constrains the actual everyday teaching and learning in our schools that's the only way we can think about what it means to improve the experiences of young people in our school buildings. So before releasing those on the escalators I would ask them isn't there really a bigger share question we're all grappling with his educational researchers. And isn't it something like this how do you get every child to feel and to be successful in the ways defined not just by basic skills test but also by their communities by the future of the job market by the demands of contemporary life by the demands of contemporary social problems and how to get every child ready to contribute all of his or her gifts to society. And as I would release my minions I would remind them that good educational research I think is that which would set forth collectively to answer that question. Thanks.

15 minutes. Please. Or get a question for Dr. White. I spoke about the academic. I wonder if there are any conversations whether there are other things there. Certainly there are many things in being in schools and academic achievement does not only help the sorts of scores kids get on the test but as they progress through school whether they graduate from high school whether they are prepared for an inner college and whether they

persist so it's not just basic skills test or state assessments. There is a question and one that would require an answer by anybody who holds my position and that is with 250 million dollars a year to spend. Are you going to focus or not. Quarter of a quarter of a billion dollars sounds like a lot. The Department of Health and Human Services spends 44 percent of its discretionary budget on research. The U.S. Department of Education spends less than 1 percent of its discretionary budget on research and investment of a quarter of a billion dollars in knowledge generation. Given the size of the education economy and the problems to be sought as a trivial level of advancement and I don't think we're going to generate answers that sustain the enterprise that

produce more funding create greater appetites for research and less we focus. And so that was the nature of the priority setting enterprise it's not to say that other things aren't important they are. And I think we'll get to a point in the education research enterprise where there is sufficient funding and sufficient stability of the enterprise itself that we can expand the scope of that and look for an L. A judgment was made primarily my judgments and so I put it forward as a proof that we would focus on academic achievement. Try to produce it not some homework so some doubles and triples to elevate the enterprise in the sense of perceived importance to its customers and will allow us to move forward into which more flowers can.

And being here and that's my understanding. So one of the frustrations that I hear both among students and also amongst faculty members is this idea of sort of generating and I think I was well taken. Important research that is actually connected to the field and it connects to policy and not really having a voice in those discussions. And I understand that it is the responsibility of educational researchers to make accessibility of other research but I'm also wondering about the responsibility of the government and other institutions of higher education. And even as individuals sort of the interaction between both the government and the researcher or the public and the researcher and sort of how does is it simply a one way street in terms of making more accessibility. Are there things that can be done on the other side to generate and I guess also to be more open to research and I look to your

thoughts on that. Well actually it's my art. One way you go to try to consciously get there was to discuss the problem. Yeah I think I know what I was both. The hardest work to do. We're trying to identify him

mobilize a handful of big question for the country serious attention to the debate with him. But I trust that if you did a good job of organizing the questions that people would bring their own theories and their perspectives to that discussion and where we would come out maybe not with anyone. Well we're relatively more of what questions were at least a few powerful ideas about what we just agreed to you know.

So I must you know I think that you know like in all fields the few bad voices there for now will organize it locally. You know it gets pretty noisy inside. So you know we have I think a role to play to kind of foster that you know as well. But then just just go and.

I figure there's a big difference between hard and organized debate about the things we know in the right side of that issue but we this is a time when we could use a lot more of that. Let me just add to that that I think there are two at least two things that need to be that need to be done. One is that and I'll speak to the federal government's role here. We need to do as good a job as it's possible to do in disseminating what bubbles up from the research community in ways that. Are in it with products that are consumable by practitioners and policy makers researchers write for researchers it's almost impossible to get researchers to write for a different audience and do so. And do so well and of course it's not just writing it's Friday dissemination products so we've been spending a

lot more attention on that issue. And I guess if you haven't seen our practice guides we have three out invite you to take a look. They are written for practitioners to try to distill. What we know address is systemic problems for example how to educate English learners in elementary school how you can organize instruction around what we've learned from cognitive psychology are two examples we got one coming out of a dropout prevention in a little while. So producing products that are accessible and consumable understandable is very important. The other part of solving the issue of having two communities talk at the center nation is partnerships with practitioners so that rather than work bubbling up that we hope is relevant to practitioners we're working with practitioners to solve their problems and the dissemination route is very short. In

that case because the problem itself is one that's defined by the community that the researchers are working with. We've established the Urban Education Research taskforce. We fund. Well-read well represented on that task force or our superintendents we've been meeting with superintendents around the country and the goal of this enterprise is to generate. Problems that emerge from urban school districts that we can coalesce a set of researchers around and funding around in order to in order to answer so that's very important to getting the research community in a place where practitioners find its work its work relevant. A challenge there is that a lot of what practitioners are interested in solving or addressing is in many cases not as intellectually as engaging as issues that the Academy defines as as important. And I think ultimately those problems are interesting but they often don't come

packaged with theoretical perspectives that allow them to be seen immediately as interesting. Which is that. Inherent in the project of a sort of enforcing a certain mode of inquiry or enforcing a certain image of what is imperial or measurable is that only some questions in the end will be will be investigated and will be answered. So I want to just keep on the table I think that people and questions can bubble up from schools and from communities but if only some of that can be investigated using hypothesis testing in a number of them won't be answered and I think that that's something that we just need to think about. Well I have a sense I think that's directed. No I think it is yes we want you know to try to do that. The notion that there are there is only one method defined as

it is created out of whole cloth at least in the whole cloth as it is not one that you can rate my state. Between methods and question. It says very clearly that researchers need to box needs to have a variety of tools and the tool that selected needs to be the one that's appropriate to the question that's being and not a method that's appropriate to answering the question of whether an assessment can be produced that's reliable. It is not the method that can be used to determine whether the relationships in large scale databases are interesting and suggestive. Educational pathways that need to be explored. It's not the method that can be used to observe classrooms and figure out what's going on in interactive classroom strong methods they have to be applied to the appropriate question.

The previous dean of the School of Education told me his story the last time a graduate student came in early in her tenure of questionable research topic asked her what she wanted to do research and she said qualitative methods. I want to use qualitative methods. That's like. Somebody in the construction business saying I want to use a hammer. The method has to be appropriate to the question. And when you start with a question the methods that are appropriate for that question and rigorous with question are fairly easily understood again within within the community of research on those issues. However if you're interested in determining what works if that's an ultimate goal I can tell you from a practitioner point of view it should be a question of what works is best answered with randomised trial

and that's why we try to move the field using a variety of things to a point at which it's developed interventions and programs that have the potential of affecting student outcomes. And when you're asking that question the best way to get an answer to policymakers practitioners in the rest of us can stand is to do right. So that's not suggesting that randomized trials are better than anything else it's adjusting for a particular sort of question that is terribly important to the practicing policymaking community that that is a method which there is general consensus about in the community of science provide some strong answers next. So thank you America and I appreciated hearing thoughts about where the field of education stands and I think actually this is sort of

more directed to it. You made a comment and I think that even echoed some of the statements that when you came into your positions in the Department of Education that it felt like those in the field of education was a little bit adrift necessarily out of focus and said that it had drifted from the procedural norms of science and maybe drifted more into modern and humanistic thought and you know as as a researcher who's developing my own epistemology and my own thoughts about the world that concerns me a little bit when I start to think that funding decisions may be experimental design and measuring for effects isn't the only answer but when that's what's being funded it starts to feel to me like that's what I'm being told as a researcher that is the answer so I'm just curious to hear a little bit more about what you meant by the field

drifted into post-modern and mystic thought and why necessarily procedural norms of science are the way to go. Well if you run something that's supposed to be. That occurs because of the Education Sciences reformatting then how can you responsibly manage such an agency without having it conduct science and science is defined by the norms of the science community. I. Don't know what else to say about that. Now with respect to what I asked funds. We fund. But if you look at our research you get say in terms of methods it is shaped like a triangle and at the base of the triangle are our goals of cation and development. Seventy five percent of the research we fund is in those two categories neither of those categories involve experimental methods. In fact we have to suppress every

year people who submit grant applications under the development goal and think they've got to do an experiment to get funded. And our most recent funding announcement we've said you can't describe an experiment we won't fund you if you describe one we're not interested in that we're interested in you developing. Developing something at the apex of the triangle far fewer than some people think we ought to be funding at the apex our studies that use randomized trials or I call it cause the experiment is on. Again if I go and it's me it's a measured go by the Office of Management Budget is to create the interventions that work to increase human and increase student achievement. You have to use that method at the end of the development process to find out what works you are quite free to use a variety of methods to develop materials and to explore what's out there to develop hypotheses. There are clear distinctions between the way scientists to do their work in terms of their disciplines in the way that humanistic and.

You can wear both hats or you can wear one hat or the other. But if you want to play in the community of science then you have to operate by the norms that are established in that community. You can get tenure for certain sorts of behavior and the Department of Biochemistry and those same behaviors would not produce tenure for you in the Department of History. It doesn't mean that historians are more or less worthy than a biochemist. It does mean that there are a set of norms that are established in disciplines. And so that's what I know that the Institute for Education Sciences operates that tries to operate within the broad norms of science as they apply to human behavior. I would want to just weigh in before finishing. The fact that when we say what works I think that still is in something of a closed loop sort of allowing only certain measurements of working. But also I want to put a circle around the word science

because anthropology and sociology are social science. These are they demand inquiry into real time phenomena into interactions over time. Ethnography for example which is a social science method is does not always it's not always an increased pathway that involves hypothesis testing which is how you labelled what science is. And in a sense under that definition I think my first book which won the award wouldn't have been funded. And so I think that. We need to keep questioning what exactly we mean by science as well. So Mike has the last word on that and I just wanted to say welcome to everyone. Jenny Wiener I'm proud to be serving as one of the tri chairs for this year's student research conference a 13 panel with Meredith Mira and Leah and I just wanted to seven special thank you to our guests are Stan Getz Dr. Whitehurst Dr. wherein Dr. Pollack and the Dean as well. Thank you so much. And I just wanted to welcome all of you our conference is running tomorrow from 9:00 a.m. to 5:00. It's 5:00 p.m. It's open to the

public and we have a a booth outside with all of our many panels for tomorrow. We're proud to invite and and to have so many student researchers from laws across the country. Our current numbers 275 people are registered for the conference. So this will be the biggest year ever. And we're very proud and very happy that you could make its nights. Oh thank you so much and have a wonderful evening.