Lies, Damn Lies, and Survey Results

Each step was oh so reasonable. But the end result? Wrong!

A group at the London School of Economics recently published a study on What Do CEOs Do as part of their Executive Time Use Project. The study was focused on internal vs. external meetings. It also categorized time spent into a couple of other categories. Just another survey among the hundreds of thousands looking at business activity.

The Wall St. Journal wrote an article about the study, “Where’s the Boss? Trapped in a Meeting” listing out the results as well as interviewing several CEOs to get some on topic quotes. Again, totally normal. And in the article they included a very catchy donut chart (charts are great because they attract attention and take up space.

Julie Kmec picked up on one part of the graph and wrote a blog titled “Where’s the Boss? And What Counts as “Work”? And from that she delivers this giant criticism of CEOs.

I’m more interested in the task that occupies the greatest amount of a CEO’s time in a typical week—the 20 hours of “miscellaneous” activities. The fine print indicates that the “miscellaneous” activities include time spent travelling, in personal activity including exercise or lunch with a spouse, or in short activities like quick, unscheduled phone calls.

Without this personal time, a CEO’s average work week—35 hours—looks closer or shorter than other workers.

If this is true it’s a giant indictment of CEOs, where they get paid significantly more than those who work for them – and yet put in less hours. If…

This is counter to what I have seen at every company I’ve worked at. My initial reaction was that it might be the industry or company size. Sure enough if you read the study, the survey sample is:

Participants to the survey were drawn from a population of 349 CEOs set to take part in an executive education course at the Harvard Business School in January 2010.[6] Prior to the program, each participant received an email invitation from the leaders of the executive education program, providing a link to a password protected website which allowed participants to fill in their time diaries online.

Out of the initial population of 349 individuals, 107 responded positively to the invitation. Of these, 42 observations had to be dropped as the records were incomplete (i.e. less than 4 days were recorded), inaccurate (i.e. the activities description was incomplete), or the respondent was not a CEO. The estimating sample thus consists of 65 CEOs observed for at least 4 complete days.

[6] The program typically attracts smaller, privately-held firms that are headquartered in locations around the world.

The survey was of 65 self-selected individuals who took a single course at HBS. To say this is not a random sample is an understatement. It’s CEOs who have enough spare time to spend a week in a class (i.e., none will be from the start-up world). Ok, so this says nothing about most CEOs but at least it tells us that CEOs who spend a week at a HBS class have a really relaxed work schedule – right? No. Let’s dive into that miscellaneous category.

The study never uses the words miscellaneous, exercise (in the meaning of the graph), or personal appointments. And the use of travel could well mean business travel only:

The survey also asks to record the total time the CEO spends in activities that last 15 minutes or less or in travel.

That 20 hours of miscellaneous is very likely 100% work time. The survey was focused on measuring activity and attendees in meetings so they threw most other activities, including shorter meetings, into the miscellaneous category. Reasonable thing to do for the study.

But the Wall St. Journal then described that “Travel, exercise, personal appointments and other activities.” This totally mis-construes the study results. Keep in mind that this time is any activity in this category takes under 15 minutes. While I would love an under 15 minute work-out, there’s no way that makes sense. This is total bull-shit.

Julie Kmec follows the example of the WSJ and turns this category into:

travelling, in personal activity including exercise or lunch with a spouse. … Do we count travel time to and from a job as “work”

Really? Lunch with a spouse was one of the listed activities? Travel to/from work was listed as part of the travel category? No. Ms. Kmec is a professor and yet clearly only looked at the WSJ article, didn’t both to make one further link to the original research, and then added in additional activities. In her case, based on her areas of research, I’m guessing she was thrilled to find data that met her pre-conceptions and doubled down on the descriptions.

Ok, so maybe there was additional documentation listed that is not in the published results. Maybe all these personal activities are in the raw data. So I reached out to the researchers and the WSJ reporter to see if there was additional information. My email and the two responses are below. Raffaella does list additional areas for the miscellaneous category but uses the phrase “may include” which, back when I went to College, wasn’t how one described data. (Yes, the Large Hadron Collider “may have” seen some particles travelling faster than light.) And provides no specifics from the data to back that up.

And Rachael provides no specifics which leads me to believe that the wording in the chart was made up and the easiest out is her vague reply.

As to how much time does the average CEO spend working? Got me (although in the software community I’ve found 55 hours/week to be well under what almost every senior executive works). But we’re not going to find an answer from research, news articles, and blogs like these.

From: David Thielen
Sent: Monday, April 30, 2012 7:12 AM
To:
A.Prat@lse.ac.uk

Cc:
rachel.silverman@wsj.com; rsadun@hbs.edu; O.Bandiera@lse.ac.uk

Subject: RE: Where is the breakout of misc?

Hi;

Thank you for the links. I read both of those and did a keyword search. The phrases miscellaneous & “personal appointments” do not appear in the documents and the phrase exercise exists only as a different context. I understand the 20 hours miscellaneous having the description of “Travel and other activities.” But where did the addition of exercise and personal appointments come from? Is this from the research or was this added by the person making the graph?

And is “other appointments” accurate? From reading the documents the description is “activities that last 15 minutes or less.” So wouldn’t a more accurate description be “short activities.” The “other” gives the impression that it is for activities other than meetings, phone calls, etc. when a lot of that may be very short phone calls, meetings, etc.

I am asking about this because that 20 hours is being used as an argument that it is all personal time and CEOs only work 35 hours/week. That argument to a large degree revolves around the “exercise, personal appointments” description.

??? – thanks – dave

From:
Rachel.Silverman@wsj.com
Sent: Monday, April 30, 2012 9:49 AM
To: David Thielen
Subject: RE: Where is the breakout of misc?

Hi David, this is what the researchers told me. Cheers, Rachel

From:
rsadun@hbs.edu

Sent: Tuesday, May 01, 2012 6:53 AM
To: David Thielen
Cc:
A.Prat@lse.ac.uk; O.Bandiera@lse.ac.uk; Rachel.Silverman@wsj.com
Subject: Re: Where is the breakout of misc?

Hi David,

Thanks a lot for your interest in the paper.

Our survey measures the beginning and the end of the CEO work day, as reported by the Personal Assistants (PA), plus detailed information of all activities lasting more than 15 minutes excluding travel and personal activities occurring in this timeframe.

Therefore, the 20 hours classified as miscellaneous may include different categories of time use:

a) Work related activities lasting less than 15 minutes

b) Travel (to and from the office or to meetings)

c) Personal activities

Unfortunately, in the specifics of the data reported in the WSJ, we are not able to know the relative importance of a vs. b and c (we have since improved the methodology, which has allowed us to measure with better precision the miscellaneous category).

I have not seen the blogs you refer to, but in principle using this data to conclude that CEOs only work 35 hours is likely to be incorrect. First, as I mentioned above, some CEOs might run several short meetings (i.e. shorter than 15 minutes), or they might work while traveling. Second, they might work in hours when the PAs are not on duty – e.g. early in the morning or very late at night.

I hope this helps. Do not hesitate to get in touch should you need additional clarifications.

Raffaella