Millot: Professional Ethics on Holiday from EdSector?

We believe that by marrying the methodological rigor of sound research with the communications excellence of the best journalism and the real-world impact of policy analysis Education Sector is uniquely positioned to both make a compelling case for fundamental reform and to promote change directly with policymakers.

From EdSector's Our Mission and Strategy

How could EdSector change Tom Toch’s critique of CMOs in Sweating into the pro-CMO rhetoric of Growing? By ignoring the professional ethics that support “sound (education) research.”

Universities and research organizations maintain formal ethical systems for their research staff and managers. These consist of a written code of behavior, procedures for the review of materials intended for publication, and a process to investigate violations. EdSector appears to have no system to define or enforce professional standards. Like many small nonprofits, it relies on informal norms, the integrity of staff and management, and oversight exercised by its board of directors. EdSector could only publish Growing  because this approach failed on multiple occasions.

Although EdSector lacks a written code, the nonprofit has stated a commitment to the norms of education research. Consequently, it’s reasonable for consumers of EdSector’s reports to assume the organization acts in ways that are reasonably consistent with ethical standards developed by the American Educational Research Association (AERA).

Over the next few days I will draw on AERA's code to offer examples of EdSector’s ethical shortcomings. The sections most relevant to my discussion are I. Responsibilities to the Field; III. Intellectual Ownership; and IV. Editing, Reviewing and Appraising Research. Readers can download the code and consider the matter for themselves.

I. Responsibilities to the Field

The first standard in this section states [e]ducational researchers should conduct their professional lives in such away that they do not jeopardize future research, the public standing of the field, or the discipline's research results. The second defines actions that jeopardize the profession: Educational researchers must not fabricate, falsify, or misrepresent authorship, evidence, data, findings, or conclusions. 

• There can be no doubt that EdSector misrepresented authorship. Growing identifies no author, although Tom Toch drafted the initial draft, i.e., Sweating. We now know that EdSector Policy Director Kevin Carey rewrote the report, but he does not claim exclusive authorship. EdSector now implies that everyone listed on the page that typically identifies an author and those supporting publication share authorship, but that is not apparent from the page itself.

• Evidence, data, and findings have been misrepresented and, when taken as a whole, falsified by Carey as he rewrote Sweating. Fellow TWIE contributor John Thompson sums the matter up in his comparison Sweating and Growing.

(Referring to Sweating) But Toch then writes "the research for a report on CMOs that I’ve produced for the think tank Education Sector reveals that many of these organizations are going to be hard-pressed to deliver the many schools that Duncan wants from them." (Referring to Growing) Education Sector sees the same evidence and writes "state and national leaders increasingly see leading CMOs as an important part of their larger plans for educational reform in the toughest educational environments." .... (Toch's) Education Sector research duly reports the disappointing results of its favored reforms, but to the Education Sector spin masters, those shortcomings are not as important as the ambitions of "reformers." And it may also be illustrative that the Education Sector concludes by praising "the core mission that unites all leading CMOs" while Toch ends with a plea for collaboration between charters and public school systems.

No original research was added by the policy director to produce Growing. Carey did add original conclusions, but he supports their emphasis by removing or ignoring Toch's conclusions and subtracting important content from Sweating.

• Growing’s conclusions constitute a misrepresentation or falsification of Toch’s draft. While they might be plausible based on evidence not presented in Growing, in Sweating Toch argued that the kind of political conclusions subsequently inserted by Carey do not address the fundamental business problem identified by Toch's research. Moreover, rather than address the evidence behind Toch's key conclusions in Sweating, Carey simply ignored it by disposing of it.

The tenth standard requires researchers “to decline requests to review the work of others where strong conflicts of interest are involved." 

As reported by EdWeek on December 3, Toch contends that Ms. (Kim) Smith joined with the NewSchools Venture Fund in lobbying heavily to soften the report’s depiction of CMOs’ financial fragility and to remove data gathered on the 17 charter networks that NewSchools has helped support.

I am not sure if Smith considers herself an education researcher, but to the extent she does, she was under a professional obligation to decline the opportunity to become what a NewSchools spokesperson called one of the official peer reviewers of the paper. Ms. Smith coined the term CMO, developed an investment strategy to pursue it, and headed an organization that led investment in many CMO’s. Smith’s status defines a conflict of interest in research.

Whatever her status as a researcher, Smith is also subject to an entirely separate set of legal obligations as an EdSector board member. She knew, or should have known, that her role at NewSchools constituted a legal conflict of interest per se. As an EdSector board member, she was in a position to exercise undue influence over the editorial process. A responsible board member with Smith's conflicts would recuse herself from any review of CMO reports. (A prudent board, informed of Smith's intent to become a reviewer, would insist on it.) Merely by participating in the review process, Smith cast doubt on the credibility of any final report, and so acted contrary to her duty of loyalty to EdSector.

Ironically, as a board member of NewSchools, Smith has also violated her duty to that organization. Deciding to serve as a reviewer and appearing to influence a report on NewSchools' CMOs can only call into question the viability of NewSchools' portfolio and the credibility of its management. Rather than defend Smith's role, NewSchools' statement should have recognized the conflict of interest and offered some apology for its employee/board member's misstep.

(It's a very tangled web.)

Next: The Ethical Difference Between Intellectual and Legal Ownership.

Note: A great many TWIE readers remember edbizbuzz , my Edweek.org blog, and have welcomed me back to the edublogfray. Thank you. Alexander reintroduced me by referencing some edbizbuzz posts relevant to my current series. He also interviewed me some time ago and, while my teenie-tiny firm's products and prices have evolved, the transcript offers my new readers a sense of what I have at stake in the Sweating v. Growing controversy.

Comments

[Susan]

Four Questions:

1. Both Toch and EdSector say that EdSector was going to credit Toch for his work, Toch asked to have his name removed, and EdSector complied. Should EdSector have declined?

2. Is this research or journalism with analysis? If it is research, what is the formal method, on what basis were the interviewees selected, what does the sample represent, etc?

3. How did you acquire the confidential review draft? If it was given to you, under what terms was it given?

4. What responsibility do you have to inform your readers about your long history of unsuccessful litigation against many of the protagonists in the report?

http://www.edweek.org/ew/articles/2003/12/10/15charter.h23.html

http://www.edweek.org/search.html?src=63&advanced=false&qs=charter+millot+suit+&qs1=charter+millot+suit+&qs2=&qs3=&qs4=&prd=a&occ=p&prx=p&srt=r&go=+++Go+++

[Millot]

Millot Answers Four Questions:

1. Both Toch and EdSector say that EdSector was going to credit Toch for his work, Toch asked to have his name removed, and EdSector complied. Should EdSector have declined?

ANSWER: I will discuss how the code addresses this later in the week. To telegraph the principle involved, there are two distinct issues: One is the appropriation of Toch's work. The second is the fraud perpetuated on the readers of Growing; e.g.,not being informed of the full circumstances under which Growing was presented to the public. Had the story now being uncovered been described in Growing, readers would understand what they were reading. I think its obvious why that story was not included.

2. Is this research or journalism with analysis? If it is research, what is the formal method, on what basis were the interviewees selected, what does the sample represent, etc?
'
ANSWER: As noted at the top of today's post, EdSector presents its work and reports as a fusion of research, journalism and public policy. As such its subject to the norms governing excellence in three disciplines. I didn't set the bar.

As for the method, as a former Senor Social Scientist at RAND, I was involved in a number of research projects based on real but non-quantitative methods. The "case study" is a recognized form of research in soft sciences like education. There are is a small universe of people in "C-level" management positions at New Schools' CMOs. I can't speak for Tom, but a statistician would call his group of interviewees a "purposive sample." Tom spoke to a good number of them. Tom will have to explain the circumstances under which he interviewed these folks, and the rules of the road. However, he has not been accused of getting the interviews - or the financial data, under false pretenses.

3. How did you acquire the confidential review draft? If it was given to you, under what terms was it given?

ANSWER: No comment, except that the question is a diversion. No one has denied that it is the real article.

4. What responsibility do you have to inform your readers about your long history of unsuccessful litigation against many of the protagonists in the report?

ANSWER: I'm not sure about "long history of unsuccessful litigation." I do have an old history - a dispute with the now defunct National Association of Charter Schools settled out of court some years ago. Neither EdSector nor NewSchools, nor any of their board members, officers or employees were part of that dispute.

I am well aware that anyone can Google "Marc Dean Millot," so there's not much point in trying to hide what's in the public record.I believe Alexander and I have made reasonable effort to give readers context for my views. When Alexander introduced me perhaps ten days ago, he noted posts from TWIE referring to posts from edbizbuzz that addressed this matter, and my resume. In today's posting I noted several other places readers could review my past.

[Sam]

Millot's answers are bizarre. Wouldn't it have been a violation to publish the paper under Toch's name against his will? And isn't it a violation to publish a second-hand pre-publication review copy of a paper labelled "DRAFT. DO NOT CITE. DO NOT DISTRIBUTE"? Of course there are case study research methods, but that terms isn't infinitely elastic, and doesn't seem to apply here. Or does Millot think that every piece of journalism is "research"?

Meanwhile, Millot's failure to disclose to readers that he was forced out of a job by and subsequently sued (unsuccessfully) some of the same funders that he now accuses of exercising undue influence tells you everything you need to know about his integrity. Surely he and Alex don't actually believe that "you can Google me" is an adequate standard of transparency and disclosure?

[Millot]

Millot Answers Sam:

ANSWER: I'm not against answering questions related to AERA's code, but we might learn more if readers read the code and made their own case for whatever position they are taking rather than merely infer some problem with my interpretation.

Wouldn't it have been a violation to publish the paper under Toch's name against his will?

ANSWER: Yes it would. And as will be explained in my December 8 post it was a violation to publish Growing without noting it's debt to Sweating.

And isn't it a violation to publish a second-hand pre-publication review copy of a paper labelled "DRAFT. DO NOT CITE. DO NOT DISTRIBUTE"?

ANSWER: If the draft was a fake, yes, but its the genuine article. And if exposing fraud is unethical, well, you can finish that sentence for yourself.

Meanwhile, Millot's failure to disclose to readers that he was forced out of a job by and subsequently sued (unsuccessfully) some of the same funders that he now accuses of exercising undue influence tells you everything you need to know about his integrity.

ANSWER: My history with the Association is a matter of public record. Everything related to that issue was settled out of court. I have made no accusations in TWIE about any of the funders involved in that dispute. The only organization I have implicated other than EdSector is NewSchools. Neither was a funder of the Alliance. Moreover, I have accused NewSchools of nothing, I have only pointed out that one of its board members, Kim Smith, should not have been selected to review Growing and should not have accepted

Finally, killing the messenger isn't a very principled defense. Argue the merits man!