Pseudo-Science or Useful Tool?

Leave it to Gardner! I’m glad he’s coming back to Virginia.

A great complement would be the ability for the class to share digital content (articles, books, videos, bibliographies) with other within the discipline or with others taking a similar course in other universities. Academic Room may make that possible one day: http://www.academicroom.com.

No disrespect to the librarian or the faculty member involved in this innovative, collaborative effort, but the use of the moniker “information concierge” reflects a fundamental problem in academia: that librarians are viewed as being on-hand to fetch information for students and simply hand it to them the way a concierge would fork over a toothbrush to a hotel guest who lost their luggage. Instead, the role of the academic librarian should be perceived (and promoted) as instructive. As an academic librarian my role, and that of my colleagues, is to teach students to navigate, evaluate, and use information effectively and ethically on their own. Seems like that would be difficult to accomplish while attempting to help an entire class at once in real-time via Twitter. But perhaps Ms. Filgo is a fast typist.

Though I have served as an “embedded librarian” for many years and recognize the value of such a service when appropriate, I am left wondering: if this class meets on campus, why didn’t the librarian simply conduct a library instruction session for the group in-person or online, then follow up with students individually on an as-needed basis using Twitter, Facebook, F2F appointment, email, chat, etc.? That sounds like a greater recipe for success (in terms of conveying the knowledge and skills to research successfully on their own).

A main goal of this is to break down the invisible barriers between students and reference librarians. It may not “scale up,” but as an occasional course infusion for selected classes I think it’s a great idea.

Of course you couldn’t scale up, but even if you could you wouldn’t want to; there are many classes for which an embedded librarian wouldn’t be helpful or necessary. Better to target courses like this one that make heavy use of library collections and resources in assignments and discussion. Then follow up with individual students on their research topics. Yale’s Personal Librarian Program (http://www.library.yale.edu/pl/) is designed to work this way. We’ve found the PL model to be more effective than offering a library instruction session (because students’ topics are too diverse or imperfectly formed, they learn little in a general session) and a more efficient use of librarians’ time.

I’m assuming that the students communicated about the course content outside of class via Twitter as well? If so, one way to scale up (while also scaling back?) would be to add the librarian to the Twitter discussion without the real-time in-class component. In that way the librarian could communicate at his or her convenience with the group or with individual students, and could potentially juggle more than one class at a time. Embedding a librarian in a course blog or discussion board is a similar option. I love this example though – very innovative. Keep up the good work!

Information specialists using social media outside the classroom to reach today’s students and support the curriculum can add much pedagogical value.

This just popped up in my gmail and already there are comments from 3 days ago!?!? My IM and email need geritol. But not this subject. I have forwarded it on to other Reference and Emerging Technologies librarins. They should get it befoe week’s end.

Utilizing the expertise of those individuals w/library science degrees is of great assistance to students navigating through Higher Ed.

Instead of worrying, Ellen, I suggest your just helping similarly when you can. Providing too much can promote complacency. Word gets around about exciting, sporadic happenings. Work well done will benefit someone somewhere.

You could reveal a bit about how you were able to do this…perhaps having one student (for assignment credit or such) shadow/observe your preparations/work behind the scenes to get a bit of a sense of how and where resources could be found and that librarian expertise can be crucial. Preparation prior to this kind of class could be done by students to reveal what they need/want to learn.

Some other students might well benefit from your extending this twitter help a few steps further. How about trying that same format at the reference desk? You could try this twitter reference class in place of a library orientation class; in the next class the students could extemporaneously reveal their reactions and explain how it might have been possible (with your gentle expertise as a guide when asked.) Expect that some might find reference you might not have considered relevant/important! They can tell you what more info they feel they need re the library resources and services which can engage their learning for a subsequent class either in their own course or another on campus in which they might ‘sit in’ for this.

Thanks Jeff, for passing this along. Kudos to Ellen and Gardner for engaging learners early so they can choose their paths to knowledge. Wait too long and they can become students who just want to ‘get an assignment done.’

twitter = tremendous possibilities

As a secondary school/public librarian I vouch that “early” Information Literacy is never early enough! Elements of it should be introduced when a teacher asks a student to gather information, which can be as early as 4th grade, and students are generally sent directly to the Internet to gather information. More apt teachers do give some guidance and links for this kind of exploration. Some awareness of online sources available through public libraries (e.g Britannica Junior Online) might exist but the access to them is indeed cumbersome.

It is also worrisome how many teachers, even at secondary school level in some subjects, do not press students to note the sources of their information… it does not have to be an elaborate format, just plain “who, where, when” kind of acknowledgments so students are saved from “inadvertent plagiarism”. As a token towards showing students how to avoid plagiarism, in high school the librarian gets asked to present in one session “how to cite”. I point out that MLA, APA, etc are ways of formatting, so it is showing students how to format, not cite. Generally students have a difficult time to learn overnight *how* to cite, which is an elaborate skill, among the higher order skills, the last in line — (i) how to evaluate, (ii) paraphrase… (iii) cite.

I don’t think I could teach a student how to ask for directions in a foreign language in one session and yet we expect students to cover the basics of information literacy in sporadic sessions. I agree that at university level an embedded librarian does not need to be in every class. That’s because much of the information literacy should have been “embedded” already, as early as possible.

Jeff,

Thank you for sharing. I am uploading this on my blog http://rajettan-eteaching.blogspot.com so that teachers whom I teach and the school/college librarians who visit my site could make use of your creative idea.

Okay, enough teasing.  What’s the tool?  Myers-Briggs? StrengthsFinder? DISQ? 

Inquiring minds want to know!

Let me get this straight:

The test is unspecified, as are the data for its validity and reliability. We don’t even know if it’s being used appropriately, even if it is valid and reliable.

It’s being administered in a highly stressful situation–a job application–in which candidates have an incentive to be circumspect in their responses (to put it politely).

The author (an N of 1) took it under very different, much less-stressful circumstances and is generalizing from that experience.

The example of a descriptor (“unable to complete routine tasks”) is what’s often called a Barnum-like statement because it can be interpreted so broadly. Without knowing anything about the author, I can generate some very likely additional hits with statements like, “Others don’t always appreciate your insights and intelligence.” or “There are times when you’re more emotional than usual.”

My conclusion is that the well-intended author and the well-intended HR department could probably benefit from a good course in psychometrics.

Bethel, the Chronicle editor and I agreed not to name the test.  If they give me permission, I will gladly do so.

Academic_entrepreneur, there is a very considerable body of validation for the test we use.  Since I’m not disclosing the vendor, I’m afraid you’ll have to take my word for it, but I did some looking at their literature (including peer-reviewed scholarship, not just self-dealing studies), and it looks solid to me. 

Indeed it’s true that I took it under different circumstances from those encountered by a job applicant, but since I’m not applying for my job right now, there’s nothing I can do to fix that scenario. 

Finally, the nature of the test is such that “gaming” it, and “circumspection,” are not terribly likely, I think.  This is not a test made up by “Joe’s Testing, Inc., LLC,” but a high-level test used by a large number of very serious organizations, businesses, and institutions.

How would you feel if your current job was dependent on that score?

As a college freshman (roughly 30 years ago), I took some sort of career/interest test at our career center.  It wasn’t a “personality test”.  Instead it matched your answers to the answers of successful people in a variety of fields and gave your a list of “people who provided answers like yours are working as ….”  

I was too polite to laugh out loud at the career counselor when the list showed my closest matches were Military Officer and College Professor.  I had no interest in the military and I certainly didn’t want to spend 10 years in college to become a professor (I didn’t want to be ANY kind of teacher).  Clearly, the test was junk.

By coincidence, I had enrolled in an introductory ROTC course because a) it was free and b) it filled out my schedule nicely.  To my surprise I ended up seriously considering the ROTC program.  However, I didn’t sign up.  Based on my interviews and research, I really didn’t think that I fit in that world.  I could still laugh at the career test.

I graduated and worked for four years before returning to school for a masters degree with every intention of going back into industry.  Instead I stayed in school, got my PhD and became a college professor (with exactly 10 total years in college).  I guess I can’t laugh at the test anymore.

Data is not the plural of anecdote but I’ve heard enough success stories about these various tests to trust that they have some level of statistical validity.  Like any statistical tools, they’re far from perfect, but they can be useful.

Side note:  I’m still a bit surprised that Military Officers and College Professors gave generally matching answers to this test.  Does anyone else find that an odd match?

1.  Our HR director and several other staff have training in interpreting the results.  Moreover, like ANY other aspect of the hiring process, it’s just one small component in an overall picture.

2.  Why, of course, I AM a creative genius.  Why do you ask?

Well I think it depends on your teaching philisophy TG; command and control or facilitate. Also; couldn’t it be the power of suggestion at work in your case? Unsure what to do( as most of us are at at least some point), ah teaching might fit the bill. I do not means this in any way as a failing; subconscious shaping perhaps.

“My sample size is small…”

No. You don’t actually have a sample at all. You have an anecdote.

Please correct to “My sample size is n/a…”

The confirmation bias is a well know cognitive error. It’s the tendency to focus on confirming “evidence” while ignoring non-confirming information. This particular cognitive error seems to be in play here.
As a psychologist experienced in test development, I doubt the ability of any personality “test” to provide meaningful information about potential job candidates. Extensive research by the Navy has confirmed that the Myers-Briggs Typology Inventory is not valid. The Enneagram test! Give me a break – super psuedo-science.

Not sure why you wouldn’t want to name the test . . . would be interested to hear the reasoning.  I’m with bethelcollege in being very interested to know which test your school uses, regardless of whether you personally found it worthwhile.  I’ve understood that some assessments, such as StrengthsFinder, are not supposed to be used in hiring decisions.   

The power of suggestion is always a possibility.  

I was enrolled in the ROTC course before I took the test.  I was two hours short of a full load and the course was two hours with free tuition.  That was the only reason I took it but maybe the test made me look at the class differently.

As for teaching, I was well into my masters degree (a terminal masters in applied math with a good placement record) before I began thinking about moving into a doctoral program and pursuing teaching. This was about 8 years after taking the test but it could still have been in the back of my mind and influencing my impressions of and interactions with graduate school faculty.

Everyone, thanks for your comments.  I knew this post would be controversial, and I am finding the discussion helpful. 

Several comments:

One of the sources of my skepticism about this instrument was that, like tgroleau, when I was in high school (about 32 years ago now), I took one of those career inventory tests they gave at the advising center, and despite my strong insistence that I was not interested in a job that involved a lot of math, my recommended career was “Accountant” (though in self-fulfilling prophecy mode, I now manage a budget that’s way up in the eight figures, so I guess they were on to something all those years ago). 

On confirmation bias, well, sure.  However, the results I got fit me like a custom-tailored suit, in basically every respect, positive and negative.  (And no, it didn’t say I was a “creative genius,” though it did say I am “impatient for results,” which is certainly true.)  This was pretty fascinating to me.  As a tool of self-understanding, it was certainly useful.  I doubt I’ll ever be good at routine tasks, but it was helpful to see that characteristic so clearly.

On gaming the test/giving answers that make you “look good,” again I think that the test doesn’t lend itself to that. 

As it turns out, we didn’t use it to make our hiring decision, though it was interesting to discuss the results in light of the large amount of feedback we’d received from the campus community, and it perhaps shaped our discussion of the candidates.  However, it was certainly not definitive.

Regarding not naming the instrument, I presume that the Chronicle doesn’t want either to provide free advertising or, conversely, to expose the vendor to opprobrium, either one.  I don’t know whether our contract with them requires confidentiality, but it probably doesn’t except with regard to individual results, which is, or course, an ethical imperative.

Something that interests me about many of these comments is the implicit notion that use of such an instrument somehow contaminates what was previously a Platonically pure interview process.  Given the overwhelming literature from a variety of disciplines (psychology, sociology, management, cognitive science, etc.) that indicates the unreliability of interviews to determine potential job performance, it’s hard to imagine that use of a normed, validated instrument is the hiring equivalent of the serpent sneaking into the Edenic realm of the job interview. 

Like everything else about hiring, in the academic world or elsewhere, the “human factor” (subjectivity, biases, snobbery, personality, etc.) is such a huge part of the interview process that it’s difficult to believe that what is on some level a more “objective” component of evaluating candidates is “unethical” or “troubling” in any compelling way, especially when that instrument is employed with the same skepticism and critical approach with which one should also treat letters of application, references, and other aspects of the candidates’ presentations. 

Applied mechanistically, on the other hand, this would be a disaster.  But so would hiring people strictly on, say, the prestige of their credentials. 

To be fair, there were at least a couple of other examples he used for comparison–his president and the others on campus who “have noted that their test results haven’t been quite as accurate.”

A massive problem with these so-called personality tests is that they are highly culturally biased in their assessment. People who have been raised and socialized in a North American culture will tend to conform to the norms established (and therefore the conclusions reached) by the assessment algorithms attuned to the North American context. Those who were raised and socialized in what we might call non-Western (or even European as opposed to North American) cultures tend to have results skewed in some non-normative (relative to N.A.) direction.

As context for my opinion, I am trained and certified in one of these methods (and have been exposed to many others), and this is a recognized limitation. The method in which I have been trained is specifically not a personality test, but has a focus on guiding specific interactions among people of differing “types.” It’s useful for tailoring interactions (like delegation, coaching, correction, praise, motivation etc.) so that they can be “heard” more effectively by the recipient. Even in this relatively low-stakes application, there is a big caveat for cultural norms when dealing with people from other-than-North-American origins. Even the test results are normalized by culture for the locale in which the test is administered (which admittedly doesn’t help much when a person of Asian, African, Indian, etc. origin takes the test and is scored in North America).

Part of the other problem with such tests is that people tend to “wear” the test results – internalizing the often horoscope-like characterizations to justify their habits and behaviours that too often prevent them from learning, growing, and developing. Anecdotally, (i.e., N=1, but I do constructivist-paradigm research, so that’s valid! :) a colleague and good friend has been told she’s an introvert according to MBTI so often that she often uses it as an excuse to avoid non-introvert opportunities. On the various occasions in which I’ve encouraged her to take the opportunity, she’s done splendidly well and enjoyed herself without the anxiety that should otherwise compromise a supposedly life-long introvert. Indeed, her success at shucking off the introvert label in these occasions has directly led to her professional success. She still claims she is an introvert, and has the MBTI assessments to “prove” it.

Personally, I think that such tests, assessments, inventories, and surveys are pseudo-science at best (yes, quantitative, statistics-based research can prove pretty much anything you manipulate it to prove), and individually hampering and destructive at worst.

This is a fascinating string of comments. As I join the discussion, you should know that I am a consultant who has been using a variety of psychometric instruments as aids in helping my clients to make better hiring decisions, promotion decisions, and employee development plans for more than 20 years. I am nearing completion of my PhD now, in late career. I have used a axiometric assessment tool that measures a person’s value structures as part of mydissertation research in the area of entrepreneurial judgment.

Instruments of this sort can bring insight into the hiring process that interviews, reference checks, background checks, statements of teaching philosophy, and record of publications cannot disclose. Ethical use of these instruments is never used as a replacement for these other steps in selection; it is used to augment them.Nowhere is social desirability bias more evident than in the ritual known as the interview process.HR departments know this. That’s why good ones today augment interviews with assessments.

I find it interesting that many of the comments made by others presumably have come from faculty members at schools who rely on other proxies of performance,  including SATs, ACTs, GPAs and written essays to select the students whom they teach. These measures of potential academic performance also have their shortcomings, but this does not mean that we ought not to use them.

I work with a number of instruments that are exceedingly well validated to ensure their conformity with EEOC guidelines and sound management practices. They have to be, or consultants using them–and their clients–would soon be out of business. There are many others–including scales and surveys developed by academics to prove some sort of pet model they have come up with, that seriously lack validation.

When it comes to personality tests, it is my own opinion that, for many positions, it is not who a person is that matters as much as how well that person knows himself or herself that makes the difference. Psychometric and axiometric instruments can be a powerful tool for enhancing self-awareness. Try them open-mindedly, as the author of this post did.

To clarify, when psychologists talk about “personality,” they typically are referring to general behavioral, attitudinal, and cognitive trends that we see across time.  So even though in some situations you act differently, this does not mean that your personality changes.  Take, for example, someone who is an extravert.  Certainly, even the most extraverted of people may have days when they feel a little less sociable, or situations where they may be hesitant to start conversations.  However, their personality suggests that, across situations, they will be more likely to talk, interact, and initiate conversations than someone who is more introverted.

First, I have to express appreciation for any administrator who is able to admit to prejudicial thinking based on ignorance and then go out and check.  That’s someone worth working with.  

I’ve had pretty good experiences with personality tests such as the Myers-Briggs and others, but my Chinese horoscope is fairly accurate too.  My astrological sign is also a good descriptor of me, and I think I can find my attributes well-represented in Book II of Aristotle’s Rhetoric as well.  I think we’ve been able for quite some time to delineate a range of personality types by observable habits and attitudes.  They are generalizations, however, so it’s also easy to find ways to fit a generalization of any kind, whether a personality type, astrological sign, or personality profile.  I’m not sure I’d be comfortable basing a hiring decision on a personality profile, though.     

I hope none of the anti-Bush commentators below are academics, because their remarks, particularly on Bush’s invasion of Iraq, display a painful ignorance. In 2008, the first compilation of translations from Saddam’s secret police showed a level of involvement with world terrorism which justified the invasion of Iraq far more than an invasion of Afghanistan. Saddam funded, supplied and trained most of the terror networks throughout the world, including affiliates of Al Qaeda. Typically, The New York Times devoted two paragraphs in the back pages to the story but they didn’t refute the facts of the findings.

Afghanistan was merely Osama’s landlord, charging rent to allow him to stay. Iraq was using his oil money to keep terrorist entities alive throughout the world. When doing battle with a stateless enemy, controlling real estate is less important than going after the bankers. 

We have stupidly abandoned Iraq too early, possibly snatching defeat from the jaws of victory and certainly making any reaction to Iranian nuclear weaponry – whether a peaceful or belligerent reaction – infinitely more difficult. Meanwhile, we will stay mired in Afghanistan uselessly, spending blood and treasure just to prop up the election chances of the current occupant of the White House.

The stupidity of the political and media elites is the direct result of America’s academic elites. America produces the stupidest smart people in the world. It doesn’t have to be that way.

I don’t know if W was the absolute worst President, but he has clearly earned a place in the bottom tier.

I was suckered once by the lies about a “humble foreign policy” and “compassionate conservativism”, but I sure wasn’t suckered twice. The man was an absolute disaster for the nation, and I don’t know if we’ll ever recover from the damage he caused.