Bad data bad decisions

I saw this link via DDOE to some analysis the Office developer assistance team have been doing.

They have been analysing the usage of the on-line help resources. Thats great, my concern, and the driver for the post title is in checking the validity of the dataset.

Their logic:

Office dev help is on-line by default, so they can look at what people are looking at on-line and extrapolate what ALL users are doing.

My view:

Most Office devs will install dev help locally (I have). Data about on-line usage cannot be used to infer behavoir of the population. The sample is biased and not representative.

We have already suffered due to one of these invalid data sets. With the dreadful data they used to justify the dreadful ribbon. Of course plenty of people suspect other factors actually drove that development, and the data was made up to justify it later.

I find it worrying that the unprecedented usage data that folks at MS now have is not tempered by a reality check of how representative their samples are.

In fact their fundamental problem seems to be mistaking a biased, invalid sample for the entire population.

I’m not any kind of statistician, so one of you can probably give the proper name for what I’m on about. And let me know if you agree or not.

So they optimised the UI for all the novice users who allowed the user experience program, thereby making Excel 2007 almost unusable for experienced users. And now they are in the process of optimising the help for all the inexperienced devs who went with a default install. Looks like VBA is going to be unusable in O14 (well the help bit anyway). Mind you I can’t see anyone at MS losing any sleep over that. Can you?

Do you have VBA help installed locally in 2007?

Oh and btw I think moving VBA help out of the default install is a disaster from a quality POV. Thats what it was like in 2000 and I’m convinced it was a factor in the crapness of the crappy code I saw so much of around that time. Why would they do that? I guess Microsoft really believe everyone is connected all the time. (I’m not – are you?)




8 Responses to “Bad data bad decisions”

  1. Dick Kusleika Says:

    I believe it’s simply called ‘bias’. There’s another potential problem: establishing causality. They assert that usage is in the same proportion to help requests. It may be true, but there are at least two other considerations.

    1. Nobody looks at help for the SUM function, but many look at help for pivot tables, therefore pivot tables are a popular feature and SUM isn’t. Clearly that would be wrong.

    2. Help in one area is very poor. Users go there once, find it is poor, and look elsewhere for help, never returning. This reduces the number of returning users and the number of overall help requests. The proper response would be to dedicate resources to this area – the opposite of what they’re doing.

    I don’t know if either of these are true, but they should be considered.

  2. Harlan Grove Says:

    Tongue planted firmly in cheek, I can’t see why you’d complain. Just look at how Microsoft has IMPROVED Excel’s own help system over the past few versions.

    More seriously, Microsoft very likely employs a few competent statisticians, so very likely understands potential sample bias. I just can’t accept the proposition that Microsoft is naive about anything affecting either Windows or Office. Therefore, I can’t accept that they’re mistakenly misreading their usage data. Far more likely the alternative is true: they do what they do for reasons that suit Microsoft’s interests, even if they suit no one else’s interests, and they only claim they have data to support their decisions.

    But it’s all moot. No one outside Microsoft and/or who hasn’t signed umpteen NDAs would ever be able to see their putative usage data.

  3. Dennis Wallentin Says:

    I still remember the good ol’ days when softwares were shipped with at least one manual. I still got all the books for Excel 4.0 available (including the book for MS Query). The advantage was that I could bring them with me and I also could manage to locate things in a smooth way.

    Today I use Google and books on specific subjects. I rarely use the MSFT help because I find it to be ——. My developer machine is not connected to Internet so the choice is rather easy.

    So how many of You actually find the help provided in Excel to be good?

    Kind regards,

  4. Jon Peltier Says:

    Dennis –

    Since about Excel 97, the help system has gotten steadily worse. The help search has gotten unreliable: you can even enter the exact title of the help topic, and you won’t find it). The examples have become less instructive, and the syntax in these examples is often incorrect. I’m hooked up to the internet almost all of the time, so I can look anywhere for help. I’ve found that the Excel help provided by Google to be pretty good.

  5. Simon Says:

    I totally agree, thats why I tend to install Excel2002 as well as 2003 – the 2002 help was the last one that was usable.

  6. Jon Peltier Says:

    I have 2002 installed on one computer (alongside 2000 and 2003), for my one client that uses 2002. A handful use 2000, a couple use 2007, and most are on 2003. I like 2003 the best, so that’s the only version on my main computer. Then I move to my 2000-02-03 computer and the old 2000 & 2007 laptop to make sure what I’ve just built works on these other versions. Sorry, users of Mac and 97, I make no provisions for these versions.

  7. Harlan Grove Says:

    I still keep copies of the Excel 97 WinHelp files on my PCs. HTML Help is an exquisite example of a fix for something not broken. Or a ‘fix’ that degraded help system performance.

    The two things about the newer help systems that really mystify me are the absence of an index and the inability in 2003’s Help-Search to specify that multiple words entered should be treated as an entire phrase, whereas 97’s Help-Find includes an Options… button that allows you to tell Help to search for the entered words as an exact phrase. Try searching for the phrase ‘command line’. 97 lists one topic (My Office program won’t run.) when I use the option to match exact phrases. 2003 lists 37 results with no apparent way to limit/refine the results (at least none I’ve found).

  8. Harlan Grove Says:

    I should have added that if Offices’s developers view the changes in the help system from 97 through 2003 or 2007 as ‘improvements’, then by that same ‘logic’ they may honestly view the ribbon as an improvement as well. That same ‘logic’ may also affect their inferences from usage statistics.

Leave a Reply

Please log in using one of these methods to post your comment: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: