Researchers at Carnegie Mellon University have found that “crowd-sourced” articles written piecemeal by dispersed writers stack up well against those drafted by one author.
“I am pleasantly surprised,” said Aniket Kittur, an assistant professor at the university’s Human-Computer Interaction Institute and one of the lead researchers on the project. The research team developed a framework it calls CrowdForge to split up and recombine complex, creative human tasks such as writing.
Articles created with CrowdForge rated well not only against those created by individual authors, Mr. Kittur said, but against those available on the same topics on a portion of Wikipedia devoted to short, clear entries.
CrowdForge starts with “small slices at a time and turns them into a complex artifact,” said Mr. Kittur. The framework provides guidelines for how to break down a project, assign portions to writers, and reassemble the pieces. The system also includes a method to evaluate the quality of the created product.
In experiments that led to the creation of CrowdForge, Mr. Kittur took large writing projects and then separated them into smaller tasks that were then made available to members of Amazon’s Mechanical Turk community, an online group of participants willing to work on online projects. Those who signed up were allowed to pick from tasks including creating an outline for an article, writing facts about a topic, combining those facts into prose, merging lines of prose into paragraphs, and finally turning paragraphs into a complete article. Many of the small tasks can be completed separately and simultaneously, taking advantage of a limited amount of time, Mr. Kittur said.
CrowdForge assigned the same simple tasks for any project to several people, Mr. Kittur explains.
Once the work was completed, Mr. Kittur allowed members of the Amazon community who had not been involved in the creation of the articles to rate the work based on a national rubric used for grading papers. The rubric considered the proper elements of writing, including “flow and content,” Mr. Kittur said.
“We are just starting to realize the potential of crowd-sourcing,” he added.
Mr. Kittur worked with Robert E. Kraut, a professor of human-computer interaction, and Boris Smus, a master’s student at the institute.
14 Responses to Carnegie Mellon Researchers Find Crowds Can Write as Well as Individuals
electronicmuse - February 4, 2011 at 8:02 am
This might be one the worst instances of pre-selecting the cohort I’ve ever witnessed. These researchers should try this paradigm on a typical freshman English section. At any rate, this seems to be another example of “findings” that are possibly [statistically] “significant,” but meaningless. Look into the marketplace and see if they are hiring people who can “crowd source” getting a business plan, brochure, owner’s manual, etc. written.
As is the case in any “committee,” there will usually be someone who has sufficient skill and motivation to actually get the work done. Businesses don’t hire “crowds,” they hire people.
Finally, this “outcome” begs the question as to where those who did the task “successfully” actually learned the requisite skills. I’ll bet it wasn’t in a committee.
rkdrury - February 4, 2011 at 9:12 am
Using the term “crowd source” is the only new thing here. Large, multi-authored proposals in the workplace have been written this way for at least 50 years. I’ve worked on many of them.
22074041 - February 4, 2011 at 9:15 am
Then there is the beautiful translation of the Bible performed by a large team assembled by King James of England in the 17th Century – words which have survived these 400 years.
wittseek7 - February 4, 2011 at 9:28 am
Most well-written English papers are art as well as craft. A “national rubric for grading papers” seems highly unlikely to detect the merits of, say, unusual but rich metaphor or uncommon syntax used occasionally for emphasis.
Students are not stupid. If they know the rubric for grading their work, they’ll write their essays to garner the best grades from this measure. Such an approach will mechanize and dumb down their work, whether they write singly or in groups.
I would never ask a class to write by committee. Not only would the results trend towards a lowest common denominator, but the entire project is, for multiple reasons, unrelated to any serious writing that they’ll do in the workplace.
As Electronicmuse notes above, the preselection of the cohort for the CrowdForge project invalidates the results. Carnegie-Mellon would far better spend its research dollars trying to devise computer programs that raise the inventiveness and persuasive skills of individual writers.
arrive2__net - February 5, 2011 at 2:00 am
Its interesting that team writing, using qualified writers and structured with this technology, was as competitive as it was, and there are probably a lot of contexts where that system will be productive. If you have written much you have probably experienced that flash of insight where you see that the linear, sequential model for a paper, embedded in the described technology, should be replaced by some far more original, interesting and insightful organizational scheme. Its not clear that the team technology approach could do that. Of course many projects do not have the ‘scope of effort’ that such a team effort would seem to require.
Bernard Schuster
Arrive2.net
Twitter.com/arrive2_net
jllo224 - February 5, 2011 at 8:10 pm
Although this boasts that “crowd sourced” articles will be created, the portions are enough to make them individualized. Like in a classroom, the professor gives specific guidelines for how s/he wants a paper written. This would correlate to those who create the outline for the article. Those who write facts relate to general researching for an article, and finally those who write the entire article are just like the students who write for their class.
I see a lot of parallels between “crowd sourcing” and peer editing. I can also see that CrowdForge is a lot like Wikipedia and will face scrutiny toward the credibility of their writers and editors.
mack229 - February 6, 2011 at 11:22 am
I agree with many of the comments above simply because I believe the pre selecting of the topics and also the random people were bad. Exactly what is the “national rubric for grading papers” anyways and how does it compare to other ways of grading papers? Like others said as well, students aren’t stupid, if they know what the rubric says and if they know how its being graded of course they will write their paper accordingly. As electronicmuse said “Businesses don’t hire “crowds,” they hire people” they are exactly correct and I agree completely. Who knows what kind of people were on there participating in that project.
armysfinest10 - February 6, 2011 at 4:59 pm
I completely agree with every statement being made. I feel as though the author of this article should take a step back and realize that “crowds” do not determine everything in society. If that was the case then where would we find our indiviualistic approach to everything that we do or say. As for insulting students, that really needs to cease seeing how majority of those who write these articles now can barely keep up with the youth of today. The people participating in this project just were not thinking clearly or something to that nature and I strongly believe that just because we have these so called “experiments” and test “crowds”, cannot determine the younr intellectual minds of today.
natbuck91 - February 6, 2011 at 9:33 pm
I believe that crowdforge can be a good tool to use in certain situations. For example buisnesses can have their employees use this method to divide the work that is assigned. But this takes away ones individuality. But I do believe that students should do their own work because they can lose basic writing skills. Also the fact that the people grading the papers is random is not a good method and that this whole process should be thought over.
clsatterly92 - February 6, 2011 at 9:45 pm
When I try and consider the principle of “crowd-sourcing” the first thing that jumped out in my mind was how similar the process is to a method the alternative band, The Postal Service, uses. If you aren’t familiar with this particular band this would probably be of no meaning to you so let me explain. The members of this band include artists from other popular bands as well as unknown artists. The unusual yet interesting thing about this group is the way in which they put their music together. Each member of the group is responsible for a different task in making the music. For instance someone creates the vocals, another member creates a beat, etc. All of the members of the group live across the country and they never meet with eachother to put the music together, nor do they discuss what their portion of the music sounds like. They simply send it through the mail hence the name The Postal Service, and each one of their contributions is combined into one song. Now, in my opinion the music doesnt sound any better or worse than any of the other various artists I listen to, of course it sounds different, but that doesn’t change the quality of it because the process that was used isn’t what typically occurs. Therefore, I don’t think “crowd-sourcing” and individual authors can be proven to create works that are of a higher quality than the other. They are different.
msftfan - February 6, 2011 at 9:59 pm
I completely agree with the comments that said the selection process was flawed. What the article does not address is that the people who are willing to do this study are generally smarter people. The average person does not visit the Amazon Mechanical Turk website looking to participate in studies. The first sentence on the website states “Amazon Mechanical Turk is a marketplace for work that requires human intelligence.” I believe that it is safe to say that the people who worked on this project are smarter than the average person.
nicoleann701 - February 7, 2011 at 1:32 am
While I believe that there is nothing wrong with peer review, as someone compared aspects of CrowdForge too, I don’t believe CrowdForge is necessarily a good thing in its entirity. Peer review is more of people editing a text that was written by one person, carring on their ideas and the same flow throughout the work. I don’t think that the fact that CrowdForge creates and edits texts simaltaneously is the best way to go simply because different people thing different, and I think that it would lower the quality of the overall text.
student19 - February 7, 2011 at 8:53 am
I don’t find it surprising that crowds can write better than an individual. Looking at the success of a crowd-based website like Wikipedia, and its success over the past 10 years, makes it hard to debate. Although the lack of a clear personal opinion is lost with CrowdForge, I think people need to be open to the idea of change and realize that technology becomes more powerful every day and it will be incorporated into our life.
mbelvadi - October 3, 2011 at 6:04 pm
I’d be interested to see the business model that can afford to deal with researching the “orphan works” they’re likely to get asked for.