On the Record, All the Time

Daniel P.W. Ellis, an associate professor of electrical engineering at Columbia University, conducts research on how to extract information from sound. He recorded his daily life for almost two years.

A transcript of the chat follows.

Scott Carlson (Moderator):
    Hello, and welcome to the Chronicle's Colloquy. Today we will discuss lifelogging, the practice of recording one's life through digital cameras, audio recorders and so on. Our guest, Dan Ellis, conducts research in this area. Mr. Ellis is an associate professor of electrical engineering at Columbia University, and he recorded his own life for almost two years. Welcome, Dan.

Daniel P.W. Ellis:
    Hello, I'm very happy to have this opportunity to talk about lifelogging and the work we've been doing. I think one of the things about this whole research area which is kind of different from the other stuff we've been working on is the way that it really does get at much deeper social and personal issues. And I always find it very valuable to talk to people to try and improve my own understanding of this because it's complicated.

Question from Brandon Bonneville, Independent Business Owner:
    I can see the benifits of lifeloging for historical and personal record keeping. However, I wonder how the privacy issue would be dealt with? What becomes of casual conversation when we need to worry about what we say or do at all moments?

Daniel P.W. Ellis:
    One of the things that I think is interesting is to compare what you can get with this kind of audio lifelog with what we already have, which is what you remember. The reason that memory isn't privacy threatening is because it's arguable. If I say something and you disagree with me, then that happens and it's legitimate, and the truth is going to lie somewhere in the middle. Whereas our experience with audio recordings is that they're beyond argument. The general expectation of audio recording is that it's actual evidence and you can't argue about it once you've got it.

What we'd like to do is try and figure out ways to make recordings which still preserve some kind of room for interpretation. The goal of this is not to try and trap people, it's not to try and hold people to their words, it's only for helping the memory of the user. So we'd like to think of ways of modifying audio recordings that make it more like that, and there are various things including, you know, scrambling the speech so that you can't understand it at all. That makes sure that no one's being held to things they said, but it limits how useful it is as a memory aid.

So maybe there's something in between like scrambling some of the words, so you can't look at it and say, Look, you said exactly this, because you've only got every third word. But that's still enough to jog your memory, so if you're looking at it later and trying to remember what was discussed, that would trigger the recall.

Question from Kristine Elen:
    Hi Dr. Ellis, Can you explain how people might use lifelogging in their everyday lives? And do most people's computers have enough space to store all this information? If not, where would people store it? Thanks.

Daniel P.W. Ellis:
    It's suprisingly little information, or maybe it's just that computers have gotten so big, it's just amazing what they can store. The device that I was using is just a 1-gigabite audio player. Using reasonable audio comprension, that can hold maybe 30 hours of audio -- so maybe 4 days worth. What I did in terms of storing it beyond that was I copied onto DVDs -- so a DVD would be maybe 2 weeks worth of recording and a year's worth of recording is a stack of DVDs.

What makes this interesting is that technically, it's very feasible at the moment, but what makes it useful is not quite so obvious. That's really been a lot of the motivation behind the project we've been doing, because you can't really answer that question until you've got the data that you could actually go and look at. We wanted to know what kinds of quetions you'd want to ask if you had these recordings available, and it seemed like the best way to find out was actually to have the recordings available and see what came up.

So far it's been disappointing in that there are no real compelling stories of what's been useful, but we're hoping that the tools we're developing will release some of that potential. The kinds of things that would be really nice are things like anything that you would currently take notes on.

So imagine if someone told you a phone number or made an arrangement, that information, just by virtue of you having heard it, was stored where you wanted it. And in theroy that's the information you can get from sound, but the missing part is the tool that can automatically extract that or locate it for you in an interesting recording.

We're interested in the sort of higher level of statistics that this kind of data collected on a long term could give you. So one of the things I was curious about was tracking the hours that I spend each week -- how much I spend in meetings, how much I spend doing e-mail -- because that's the kind of information you could get from it.

Question from John Doe:
    How did you ensure the privacy of people you recorded? How would others do so?

Daniel P.W. Ellis:
    It's a big problem at the moment. The only way to ensure the privacy was to basically make sure that the recordings are not available to anybody except me, so I didn't keep them online. I didn't even keep them on my laptop. I just have them on these DVDs and then if I have something specific that I want to try out, I can load one up and look at it.

The research that I've been doing has been on this single data set of one week's recording. Even the data that's been on the computer is limited to this one set that me and my student felt was not particularly sensitive.

But ideally we want to make it something that's useful where you do have more of your data immediately available and then there are different aspects you have to take care of. One is security, to make sure that if you are moving this data around, that it's difficult or impossible for it to fall into the hands of those who aren't authorized. And there's a lot of good things you can do there. One of the things we've looked at are these techniques that spread the information out between several computers in such a way that you can calculate whatever derived funciton you're trying to achieve, but no single computer ever sees the original data in the clear. There are some very nice theoretical results of when you can do that even when it seems paradoxical.

The other thing we've been pursuing is thinking about how to have automatic permission and privacy policies. What I mean is you could get permission from various people to record what they're saying -- sort of a blanket permission. But the recorder then has to know: Am I allowed to record this person who's currently speaking? To do that, we've been looking at techniques for automatic speaker identification, which is also very useful just in the other kinds of indexing or search you might want for this data.

The problem is that it makes errors, but you can always make a trade off between falsely identifiying someone as someone else and failing to identify someone as the correct person. So we can always try and make sure that we err on the side of caution by not recording people if we're not sure if we have their permission.

Question from Margo McClaren, Michigan State University:
    Nietzsche said that "forgetfulness is the doorkeeper of pyschic order." I've always found that quote interesting, and I thought of it again after reading the story. I wonder what the negative psychological impacts of lifelogging may be. What do we lose if forgetfulness is taken away from us?

Daniel P.W. Ellis:
    I think that's really cool, and you know, that may be something we can only find out by this kind of experiment. You should always be wary of getting what you ask for, because it might not be quite how you expect. One of the precedents that I find helpful in thinking about these audio recordings is my experience with e-mail. I've been sending e-mail for 10 or 15 years now, and I have records of all the e-mail messages I've sent in that time. Even though I have the record, sometimes I can completely forget that there is something there and so it effectively has been forgotten -- even though, theoretically, I could go back and find it. So even with recordings, even with automatic indexing, assuming we can make that work, you still have to believe there's something there before you find it. But I do believe that people can gain a lot by being able to reinterpret past events without necessarily being challenged. I think any kind of technological record can interfere with that, and the only thing we can do is hope that the benefits outweigh the losses.

Question from Matt Krebs:
    What are some of the positives and negatives lifelogging would have on interpersonal relationships in business/boardroom setting?

Daniel P.W. Ellis:
    I think this is interesting because already when people discuss things in a business context, it's already a somewhat artificial situation with more or less explicit rules which people have agreed to, but it's not the most natural situation. Very often, you know, business is about trust and being sure that the person whom you're talking to is going to do what they say they're going to do. So you can easily imagine situations where people are happy to have this kind of record, becuase it cuts both ways. If people are generally trying to make sure that they can agree on something, then presumably everyone is interested in an accurate description of what happened. I think there's an instinctive suspicion of this kind of technology when we don't have any experience of it. I have a little bit of experience now although I still feel like I don't understand it that well.

But my sense is that, in fact, having recordings of what we say may not be as problematic as we fear it could be. Because the fact is, people don't always say things exactly how they mean them and as data of that kind becomes more familiar, people will understand that you have to interpret these kinds of things with a grain of salt -- that data that's been collected as a literal transcript, is going to include all kinds of misstatements and incomplete thoughts and things like that.

What I imagine us getting to is a situation where maybe there's more literal transcripts available but people are rather more forgiving of the things people say. I think people are a little bit unrealistic in the standards they can hold people to about what they say. And then, you know, seeing your own recordings can make you very humble about that.

Question from Matt Krebs:
    I have found that if you write down everything that goes into your mouth for two weeks, you will really start to observe your eating habits. Has anyone ever thought of using lifelogging to help curb our nation's obesity problem?

Daniel P.W. Ellis:
    That's definitely one of the topics that's come up in this whole lifelogging research community. As Scott said in the article, there have been a number of proposals which are health related and I think that's a very powerful application area. The camera that was mentioned in the article that takes a picture every minute automatically gives you the data that you can then use to reconstruct your diet. We can certainly imagine that becoming increasingly automatic with image recognition software or something like that.

I think this points to a wider issue, which is the way that we're not consciously aware of how our memories can distort things until we are confronted with some alternative record. That's why I mentioned my interest in trying to compile statistics of how I spend my time, because I have some intuitive idea that I spend some percentage of my time in meetings. But I'll bet I'm exaggerating one way or the other, and it would just be really interesting to have that independent information.

So I think there are all kinds of aspects of our behavior that we might choose to change if only we were made aware of them, and this kind of technology can have a very positive effect by giving us that kind of information.

Question from Hassing, NYC:
    Lifelogging seems to have interesting education potential, if it helps people to be more reflective. What might be the aggregate and collective impact on society if it took off?

Daniel P.W. Ellis:
    I think it's interesting that the reason that this whole project was possible was beacuse these lightweight recorders, able to make long recordings, were cheap and widely available. We should note that the main reason behind the manufacturers' including that feature in those devices, which are mainly music players, is for recording lectures.

In my own teaching, I've sometimes taught these classes that we have at Columbia that are offered for distance learning so thatt people elsewhere in the country can watch the lectures recorded on video by the Internet. There's a whole range of issues about how that compares to classroom teaching. But one of the unexpected things is, I found my on-campus students also wanting to use those videos because they could revise the lectures, they could see a point again, and they could control the rate that the material was presented. So they could pause. And that seems to me a huge benefit.

So in a situation where information is being transferred and the rate of information transfer may not match the rate at which you can pull it out, then having the recording that you can review and control can make a big difference and can help a lot in getting the most out of the experience.

Question from Elmore Alexander, Marist College:
    I've done research involving video and audio taping of subjects. While it is clear that after a while subjects forget to a degree that they are being taped, I have a hard time imagining that the digital tape recorder and sign will not dramatically alter day-to-day interactions. What kind of research is being done to assess this impact?

Daniel P.W. Ellis:
    I thought it was very interesting in Scott's article when he mentioned about the way that he felt -- he and his wife felt his behavior changed, he was acting like he was "on stage" when he was being recorded. I didn't have that, and in fact the anecdote about me arguing with my wife is a refleciton of how I very quickly forgot I was being recorded. However, I did not have a sign around my neck. I don't know of any research being done of how people's behavior changes when they're being recorded. It's a bit of a problem because of course you'd want to record information on how they behave when they're not being recorded, which is a contradiction. My hope is that if recording becomes commonplace and everyone does it, then there won't be these two states of you knowing either you're onstage or you're not onstage. It will just be regular life, and everyone will behave as they do anyway.

Question from K. Henry:
    Hello, I'm wondering if you think someday we'll all be lifelogging all the time, and how far off that day is. (Ten years, 20 years?) Thanks!

Daniel P.W. Ellis:
    I don't feel very well placed to answer that -- although maybe I should -- but I do think about it. And I have the sense that it's not very far away simply because the technology is so relatively trivial and it seems that the benefits should be plenty large enough to make it worth doing.

The research we're doing on how to extract useful information out of these recordings is clearly a critical part in shifting that balance to make it worth collecting the data. How long do I think before we have the kind of automatic analysis that I'm dreaming about? Well, 5 years seems too short. But things can move very fast in this kind of technological area. One way that these things can work out is that rather than having a completely universal general transcription of recordings, like a human might do, instead there'll be some more narrow, focused application, which is easier to implement but still provides a lot of value. That will be enough to push the uptake and use of this kind of device.

Already one area where devices of this kind have been used is in monitoring people's media consumption, the kinds of things Neilson and those guys do. They've experimented with giving people audio recorders and then using those recordings to figure out what TV programs they were watching and for exactly how long. That turns out to be very easy because you're matching the exact sound track even though it may be noisy. Google has done some work recently on using that kind of technology as a way to, in real time, provide people with information on their computers that is related to whatever they're watching on TV. I can imagine an application along those lines based on something that is technically pretty robust that would make people really want to have this kind of recording.

When we look at the general trends of how we are using computers more and more for different tasks, I really think it's inevitable that this data, which is in theory available, will be something that we'll want to capture as a matter of course. And if we don't, it will be because society has a debate and figures out that they're losing too much in terms of privacy and the ability to forget. I think that would be a very useful debate to have. My guess is that those costs are not too great, but we'll find out.

Question from Sasha B., student:
    What reactions do you get when you're out and about and people see that you are recording them? Do they begin to act differently, and do you see lifelogging having the potential to stiffle free communication and turn us all into robotic politico types?

Daniel P.W. Ellis:
    You know, the reacitions I get are mostly pretty strong and mostly pretty negative. Maybe about 50 percent of people I discuss this with have an immediate negative reacition, which they often have difficulty really explaining. But there's clearly a strong intuitive resistance to having a more detailed record of what they're saying than memory already provides.

I agree that one of the most relevant phenomena here is the way that politicians are often pilloried for gaffes and momentary lapses. I certainly don't want that to happen to everybody. But my interpretation is that there's a degree of hypocrisy there -- that we enjoy being able to condemn certain people for not managing a standard we couldn't manage ourselves. And like I said, I think the experience of listening to recordings of yourself in almost any context makes you a lot more tolerant of what might be on recordings of other people. So I think once there's symmetry -- once everyone who is at risk of being recorded is also making their own recordings, my hope is that the discomfort and suspicion will dissipate.

The real key is what the recordings are used for. At the moment if you tell someone that you're recording them, they instinctively assume or imagine the situation down the line when you play that recording back to them, or you play the recording of them to someone else, in a way that they can't control and that they don't like. If we can address that problem technologically, then a lot of those problems can be avoided. The idea is to give people control over the recordings that they appear in, to prevent a situation where the recording could be released in a way that they don't approve of.

That's very difficult because certainly at the moment, once the recording exists, it can very easily escape control. But it's interesting to look at some similar technologies -- for instance, the way music sold over the Internet is controlled -- and to imagine using those kinds of ideas to prevent or at least limit the way that recordings can be abused.

Question from Jason Anderson, U of MN:
    Do you think a person's life logs could be used against him/her in a court of law? Or would they fall under the clause of not having to testify against oneself?

Daniel P.W. Ellis:
    Well, I'm not a lawyer, obviously, but there was this very interesting paper by Leana Golubchik that addressed that very point. From what I understood, any existing record can be subpoenaed -- that is, the court can demand you to release it. So from that point of view, you wouldn't be able to refuse to have that be used against you. I hadn't thought about that at all before hearing her talk, and it certainly made me stop and think.

In fact, she had this imagined scenario where some individual was in such a situation that they'd been accused of something, and they said, Oh, it just so happens that I wasn't making recordings at that point. And the prosecuting lawayer had said well, isn't it true that you normally make recordings, so wouldn't you be making a recording on that day?

That made me think about ways to sort of provide some sort of wiggle room in that kind of scenario, while still providing utility. One thing could be for a system which recorded randomly every other day on average -- on a given day, you didn't know whether it was recording or not. In terms of the record it made, it would be much less useful. but in terms of the legal defense, you could say, I just happened not to have that day, and that would have to be plausible.

There's also a technological solution that's been proposed for controlling records. I think legally you can't prevent a court from getting hold of this record, but you can make it practically so difficult that they wouldn't do it.

The idea is this: You have a large community of peers, and you give each of them the opportunity to lock access to your record so that neither you nor anyone else can get it. Then if some situation occurs where you wouldn't want your recordings to be available, those people can all enable a lock. Then the court can't compel you to release the record, because you no longer have control. Instead they'd have to go around and subpoena all of those people to access the record, and with the Internet that could be thousands of people.

Of course, the problem is that if any of them is malicious, or even just confused, they could prevent you from accessing your record at a random time. So it's not perfect but it's an interesting idea about how to frustrate the existing legal statutes. But for all I know, it might even be illegal to construct such a set-up.

Daniel P.W. Ellis:
    Thanks a lot for the chance to talk about this. I thought they were really interesting questions, and I'm glad we got to talk about the implications of this kind of technology.

Scott Carlson (Moderator):
    That's all we have time for. Dan, thanks for joining us for this interesting discussion.