Archive for the ‘quality’ Category

Dawn of realisation

Wednesday, 29th June, 2011

For pretty much as long as I can remember I have wondered why people in organisations persist in doing things in Excel and VBA that would be sooo much better in almost any other technology. We have discussed many of the reasons in the past but I got hit by a new one the other day:

The corporate process for requesting the right tools for the job is so long winded, painful and uncertain that no one bothers. Its quicker, easier and less pain to build an Excel monster instead.

To me this is a disaster, a dereliction of duty by IT, and just an all round FAIL.

IT should be proactively monitoring the markets and trialling interesting looking software and services with a view to proposing them to the business. Not sitting back, blocking all the interesting websites and file shares waiting to obstruct any other attempts to get the job done. (Hardly the work of a Strategic Business Partner is it?)

I have been through these processes a few times, some stuff like getting your Office install upgraded to Pro to get Access is normally a few clicks, an approval email and a cross charge. Getting approval to deploy some VSTO components and some C++ dlls at one place was a bit harder, including a half hour interview on the benefits and risks. I totally don’t mind needing to justify this stuff, but in many cases IT should already have proactively tested and approved it, imo. (or in the case of C++ understand they already have it all over)

I was looking at these processes for not so standard stuff at a few places that publish the detail, and sure enough it can be 6-8 weeks to get approval for an install. That’s ok if its bill and bobs virus ridden utility, but if its a part of what should be core infrastructure then you’ve already missed the boat.

So if you are in IT here are some tips

Developers need developer tools, they can and will develop without proper ones, but it wont be pretty – and it will be your fault for not providing the right tools.

Users need intelligent flexible access to complex data. If you don’t have the right tools, (Think OLAP/BI), then it is your fault when they do it badly in Excel.

just sayin…

What is the most long winded, laborious, misguided IT process you have had to go through? was it for software approval?

cheers

Simon

Eusprig 2011

Friday, 17th June, 2011

This years spreadsheet risk and quality extravaganza is almost upon us. 

It is exactly just less than a month away in mid July.

You can book here.

I am not presenting this year, as I thought I would let someone else have a turn speaking (and of course I missed the submission deadline).

In fact I probably wont be attending as I’m not sure where I will be working/holidaying then.

I would be expecting a good talk from Patrick as we worked together this year on a few spreadsheet related projects. Indeed he came face to face with the source of several of my formula horrors from previous years!

oh looks like he is not presenting this year, but on the bright side there is some more original research on the power (or not) of range names, amongst other interesting papers.

Here is the (current) draft outline schedule.

Are you going?

Cheers

simon

Getting the data right

Wednesday, 15th June, 2011

I was chatting with a fellow dev recently.

He said for him the most important first step (in design/development rather then analysis) was to get the data right.

I started off disagreeing and suggesting the most important first step was proving the concept will even work. But then in our discussion and thinking about it after I decided that I agree with him. To an extent anyway.

My new most important first step is to be clear how you will be able to check the data is correct. Because it may work in concept, but if you can’t prove your data is correct your system cannot be proven correct, and therefore not a right lot of use. I use prove and correct here in the traditional business consulting sense rather than legal. (correct = near enough to blame the old system, prove = no one will find out before you are out of there, invoice paid.) ;-)

We were talking about a standard, load data to sever db-run some queries-extract to Excel reports, kind of a system.

What would your most important first step be? (assuming you have some idea on what the users want/need of course)

cheers

Simon

Smurf on 2011

Saturday, 1st January, 2011

My 2010 predictions were a little tame I think, so I’m going to go more out on a limb this time around. I’ve covered a broader area this time too. I normally limit myself to spreadsheets and software, I’ve had a pop at some more general stuff this time.

IT Market

  1. Google will overtake Microsoft in market capitalisation (190 v 240 currently)
  2. MS won’t bid for Adobe.
  3. MS is fading fast as a general brand, especially with consumers, this will continue and probably accelerate
  4. Apple will pick up most of that consumer mind share in computing and software.
  5. I think someone might bid for HP this year, possibly Oracle if they can digest Sun in time. I don’t see too much anti competitive hassle from this and Larry has been softening them up with body blows for a few months
  6. Yahoo will surely disappear, I’m not sure how they survived this long
  7. Once people get bored of facebooking about facebook surely it will do a myspace?
  8. Linux won’t do much in 2011, I really thought netbooks would do it 2009/2010, but I was wrong. I don’t see a better chance in 2011. Tablets? I’m not convinced this time around.

Software

  1. Phone and tablet software will be massive in consumerland. iOS and Android (and Blackberry), not Windows phone.
  2. PC consumer software will be a non story (in the general media)
  3. MS Office is in a death spin where no one understands the value it can generate so no one invests so no one discovers. MS will continue to fail dismally to market Office. They will cut marketing spend so they will send less of the wrong messages to the wrong people.
  4. Quite a few corps will start to migrate to O2010 in 2011 as many skipped 2007 and 2003 will be 2 years out of mainstream support. They won’t leverage many of the new features though, as its ‘just Office’ not an integrated part of strategic IT infrastructure like it should be. (message/people…)
  5. I think Windows 7 may be due for deployment in a lots of companies too.

Office development

  1. We will still be undervalued.
  2. We will still be loved by users and loathed by IT, who will continue to prevent us from using the best tools for the job.
  3. Office will continue to be userland so VBA still key, although job ads will request a knowledge of C#, but then not let you have Visual Studio.
  4. Microsoft will still not have a plausible .net/Excel development story. VSTO isn’t it
  5. .net developers will continue to abuse Excel as they don’t understand the object model or native code.
  6. Access will still be looked down on as a zero credibility toy – an image which hampers SharePoint uptake as Access is a million times better for managing lists than the 1970′s web UI. I’m currently calling my Access development Jet development to avoid raising the Access Alert.
  7. Office 15 Beta 1 will probably make it out of the door before the end of the year. Expect the ribbon to be nearly as good as 2003 toolbars, lots of unusable lock-in to server components that corps won’t deploy. Beefed up power features like cluster udfs in 2010 (perhaps performant .net udfs???). More eye candy cabbage/inappropriate intra suite standardisation. Closer Excel/Access integration may be on the cards, even as many corps seek to ban Access altogether, it may cheat death once again.
  8. Plenty more vacuous ‘spreadsheet control’ projects will start in 2011, although most will be tick box half hearted affairs. The crash knocked a bit of the wind out of the compliance gravy train sails. (can I mix methaphors like that?)
  9. there will be plenty of opportunities in financial services for folks with Excel/VBA/business – this market is hotting up after being depressed for 18 months or so – so many devs will have moved on.
  10. LibreOffice will continue where OpenOffice once went as the leading Office competitor, before Oracle alienated the whole dev community. Still won’t be much of a competitor though, sadly. OpenOffice will be gone by the end of 2011 imo, at least ‘gone’ like StarOffice.

General development

  1. Java will lose some light as it has been scuttled by Oracle, I think this will cause more of a general splinter rather than a mass migration to .net for example. Ruby, Python etc will likely be winners on the web/server, but maybe C++ will find some love, it has done with MS in VS2010.
  2. Or maybe there will be an unbreakable Oracle Java?
  3. Silverlight is doomed, so VS2012 might be useable. Silverlight is the Access of UI. MS just don’t know what to do with it. Smart devs will steer clear till they decide. Gone by 2012 imo.
  4. Objective C will probably be a worthwhile skill in 2011 (not a great synergy with VBA though :))
  5. F# will get plenty of buzz, but not a lot of actual traction, just because OO is inappropriate for whole swathes of software doesn’t mean it isn’t deeply engrained (as the ‘correct/best/professional’ way of developing anything)

Hardware

  1. Well derr – tablets will be a big news story in 2010
  2. Netbooks will be replaced by tablets, in news, if not physically.
  3. The march to smartphones will make iPhone/Android/Blackberry development a very viable business model, especially compared to banging your head against the office dev wall.
  4. If the Ubuntu Tablet appears I’ll buy one ASAP.

I try not to be a ‘Microsoft watcher’ blogger partly because lots of people already do that, and partly because its hard to hit the right tone between sycophant (I earn my living in their tech after all) and irrelevant moaner.  But…

Microsoft

  1. MS is in a cost control phase at the moment. That means every investment/spend needs justifying.
  2. Plenty of possibly viable techs have been axed after massive investment over the past few years
  3. some of those have been brave (or mad – depends your pov) decisions
  4. This means very little is unquestionably safe – its perceived cost benefits, financially, ruthlessly.
  5. I think MS have lost the consumer space so they will focus more on the enterprise (which seems to work well for Oracle for example).
  6. Tight Office/.net integration? cost? benefit in increased unit sales income? – unlikely then…
  7. Windows phone? If it goes well in the next 6-12m then maybe, if not its out. Microsoft could exit this whole market with no obvious income loss (short term at least). (imo gone by 2012)
  8. Silverlight? compelling benefits or the next VB script? (imo gone by 2012)
  9. VSTO? essential .net/Office bridge? unfinishable unloved bodge? (imo VSTO will survive for a while as it joins a couple of cash cows)
  10. VBA? trusty workhorse? or poor implementation of a bad design? (imo safe as houses, there will be nothing that could be construed as a hint of retirement plans for this utterly vital (to MS) tech. Unless you take the complete lack of investment and development of the IDE as a hint of future plans of course)
  11. VSTA? remember that? VS editor in Office. Hmmm… I really don’t know if better .net/Office is coming to Office, If you pushed me I would say no, I don’t think O15 will have an integrated C# IDE.

Economics/etc

  1. No double dip recession (if anyone thinks we are out of the last one yet) just a slow (er than reported) creeping recovery.
  2. 2011 will, in line with solar physicist predictions (again), be colder than 2010 by any reasonable measure.
  3. The Urban Heat Island effect will get some attention to try to explain the gaping void between what global warming ‘scientists’ say and what normal people see and feel.
  4. We will see more climate comedy – where a council or org has made a big business decision based on promises of global warming, only to be totally shafted by actual weather. Step forward Geneva canton and Lytham council – both allegedly sold their snow clearing equipment in recent years then struggled last winter and this winter. (could we include Heathrow?)(we can take UK road salt supplies as a consistent snafu.)
  5. Financial market volatility will increase, especially on the down side as algorithmic trading gets more aggressive safety cut outs.
  6. 2011 will be even more fad-tastic than 2010, game shows, reality tv, shit music, phone apps, blankets with sleeves…
  7. No big UK bank failures
  8. Another couple of Euro zone rescue plans will be required, someone might even notice the state of the UK economy.
  9. A Euro will almost certainly be worth more than a pound, although they are both in a race to the bottom at the moment.
  10. More intrusive security pantomime mischief will make air travel even more unpleasant but no more secure.
  11. Did I mention that spreadsheet development will still not be cool?

That’s about it for now, I need to go out mountain biking now the snow has melted. Sorry this is so long!

What do you see as the big news stories of the next 12m?

All the best for 2011

cheers

Simon

Test harnesses in production

Tuesday, 30th November, 2010

You wouldn’t believe how the cold dark winters evening simply fly by here (well at least I managed to get home  even though 4″ (100mm for the metricians) of snow fell this afty, the first November neige in Geneva since 1980 allegedly). And its still dumping it down – might have to throw a sickie tomorrow and go sledging with the kids.

Anyway the big argument today was test code.

Should you or should you not put your VBA test code into production?

Should you strip down your project to the absolute minimum clean prod only code?

Or should you leave in the code you used to test your production code? (assuming you are one of the 3 VBA devs worldwide who bothers to test of course)

My vote is to leave it in, and even though I was in the minority at school today that doesn’t mean I’m wrong, yet.

If your test code is crappy and distracting then yep take it out, no probs (in fact take out all the crappy distracting code), but why I think decent test harness code should be left in:

  1. It helps show what the code is meant to do
  2. It helps when you are trying to fix something later in production at a users’ desk.
  3. It shows that someone did some testing sometime
  4. If you make significant edits to your code you should retest, if that edit was removing what you thought was the test code how will you check you haven’t broken something? add a test harness???

What do you think? what am I missing that these clean code freaks can see? Remember I’m not for a minute suggesting leaving in a load of random junk scattered throughout a project. I’m thinking of separate modules or at least sections with a bunch of meaningful tests that exercise the main functionality of the system in a controlled way.

What do you do?

cheers

Simon

European Spreadsheet Risk Group 2010

Friday, 23rd July, 2010

We had another excellent Eusprig Conference last week – congrats to the organisers.

Lots of interesting discussions both within the sessions and outside, and of course in the pub.

Some highlights:

Sumwise is a new spreadsheet-alike product that allows more structured models, runs in the browser locally or remotely. It looked really good, I can see a whole class of problems that it fixes very elegantly.

EASA presented on their tech for publishing spreadsheets to web servers for browser based usage scenarios. Having built a few of these Excel-runners myself (I still have the scars) I appreciate what’s involved. I liked the way it would work with any spreadsheet and is not as picky as Excel Services (2007 anyway, 2010 is more accommodating).

ClusterSeven were talking about new value they are discovering for clients by tracking cell changes over time. They are able to build up not just validation trends, but also business, pricing, economic etc trends. I suspect converting all that unstructured tat scattered across the average spreadsheet forest into mineable information is more valuable, and a better sales story than the hunt for ‘potential’ errors, or mischief.

Dean Buckner from the FSA described their current views on data risk, and it close relation spreadsheet risk/end user apps. I always enjoy the clarity with which Dean explains what the FSA care about and how those things should be addressed. For example sometimes just a written policy is fine, for other areas the FSA want clear practical evidence.

There was some interest in trying to create a generally agreed set of best practices, with caveats as required. I’m not sure if this is something Eusprig will officially endorse/sanction, but I think its something they must if they want to maintain credibility. You can’t spend 10 years saying ‘what about the spreadsheets?’, and then offer nothing to help.

I was disappointed to miss some of the academic papers which ran on a different track. I am not a fan of the Eusprig 2 track approach. I don’t think there are enough people interested in this area to divide further, and I think the current conf length (1.5 days)  could be extended by 3 hours to allow the academic stuff, perhaps on the Friday afternoon.

So instead of hearing the evidence of how names can impair less experienced users we had a half hour slot about why a certain modelling company use names extensively. This was a little long on hyperbole and a little short on fact/evidence for me. And it unfortunately failed to address all the real world scenarios that make many experienced commercial devs wary of names in the real world.

My favourite (repeatable) quote of the event was actually just after
Ralph Baxter the CEO of clusterseven was explaining some of their new features/use cases to me as we ascended some lift in the tube system. As we got the street level some bloke turned round and said….
Drum roll please…
“Ralph thats the best elevator pitch I have ever heard”
That bloke turned out to be Mel Glass from EASA, we all then spent the next hour discussing the harsh reality of corporate spreadsheet use. (And some of the opportunities around at the moment)

One of the people pushing for some generally approved spreadsheet techniques was Morten Siersted from F1F9. Of course we will never all agree about the minutiae (note the interminable named ranges debate). But it has to be better to have reviewed a well thought out approach and decide where you will adopt and where you won’t and the supporting reasons.
FAST is one of these well thought out approaches, and its free/open source, non commercial etc etc. And unlike some of the others, Fast stands on its own. there are no chargaeble tools required to implement or test it.

Its here.

I’m not sure where the best place is to discuss it, but I do think we should discuss it. I’ll maybe do a more in depth post in the next week and we can discuss it there, or if FAST put up a discussion blog post that would be even better.

I’m not sure which is the most contentious, climate change or spreadsheet modelling/developer standards?

We’ll see I guess.

Did you go to Eusprig? what did you think?

cheers

Simon

ps I managed to use the é and the è on my Swiss keyboard today.

Eusprig 2010

Tuesday, 22nd June, 2010

Pack yer party frocks its nearly time for Eusprigs annual conference.

In my opinion this is a must attend event for anyone serious about professional spreadsheeting. It should also be required for anyone contemplating spreadsheet management/migration/quality/control projects. Why make all your own mistakes when you can invest 2 days, a couple of hundred quid and learn from others, bypassing a heap of pain (and cost) for your own organisation/project?

There are two key aspects

  1. The technical conference content, including several schools of thought on best practice (not sure whether to bring my cornering kit, or my gum shield and boxing gloves) spreadsheet quality project post mortems, and lots of original research. And not just wishy washy stuff – proper, well designed experiments that really focus in on key issues. I am particularly looking forward to the slot demonstrating that names make spreadsheets harder to understand. I’ve had plenty of rows on that very topic.
  2. Social/networking – where else do you get the opportunity to buy drinks for big cheeses at the FSA, and big cheeses from the spreadsheet audit/management software vendors, and student bludgers? At least that’s how I remember it.

Full conference details (including dates: Thur/Fri 15th – 16th July, and venue: Greenwich Uni London, England, and agenda etc) are here.

I’m giving a keynote first thing Friday (which seems a bit harsh!).

Why not combine it with the Excel Dev Conf for the full spreadsheet-oholic effect?

Are you going to Eusprig??

If not why not? (serious question) – what would need to change for you to attend?

cheers

Simon

Excel Dev conf details

Saturday, 12th June, 2010

The draft (detailed) agenda is here.

The event location is here.

You can get yersen booked on here. Its currently 150 quid for the day (with early bird discount), that will go up to 200 at the start of July, if there are any places left (all plus vat). I didn’t work out how to do a declining stock thingy with the codematic shopping cart so I’ll just have to warn people as we get near the room limit and then disable the shop item.

Please tell everyone you know who might be interested. I think this will be an excellent event, and the first one I’m aware of that actually addresses the business of selling Excel stuff. Add in a bunch of technical content on the latest trends in corporate development, and plenty of time for questions and chat, an informal atmosphere, and I reckon we have an excellent well rounded event coming up.

If there is something you specifically want to be covered let us know. If you have questions for the panel let us know that too.

Accommodation-wise I suspect many of us will be booking into the nearest/cheapest travelodge.

cheers

Simon

Why I love and hate project management methodologies (long)

Friday, 26th March, 2010

I’m thinking of documentation focused, process map driven, meeting oriented, service level agreement based, gated review, buzzword encompassing, busy work, job-fer-life methodologies. Perhaps there are others but I havent seen them in the wild.

So why Hate?
Well, here is my ‘how to address a business need’ shopping list in preference order (fave at the top)

  1. Drop in an off the shelf system (nice in theory, rare to find a good match)
  2. Customised off the shelf system (the breeding ground for big methodologies)
  3. In house IT team developed ‘mainstream’ system (often relational database CRUD with .net/java/whatever business logic and presentation layers.)
  4. Professionally developed spreadsheet based system (pro means knows spreadsheets, knows databases, knows coding, knows software development. (funnily enough there aren’t many people who fit this bill))
  5. and last  but not least, an end user developed solution, probably in a spreadsheet (or friendly desktop database).

This list is based on my estimated cost/benefit over the lifetime.
In a great many cases a spreadsheet solution will be the quickest and cheapest, no doubt, but their lifetime is often short, and they can be a maintenance burden.

The purveyors of snake oil methodologies are mainly feeding off their big consulting firm background in implementing massively customised, big fee earning ERP systems in the job-fer-life timescales. End user developed spreadsheet proliferation is the direct response to the gross user neglect that is core part of these big systems. ‘Take the pain now, we have a brilliant system that will fix this’… in 2017! Isn’t SAP the one that no-one has actually finished implementing yet?

In a 100 dev year, assembly coded, man-on-the moon project, yeah, sure bring in a heavyweight methodology suitable for life critical systems. For a two week spreadsheet spring clean? give me a break. For modern business systems tools like .net and java etc I think the tools are powerful enough that if the initial design is good the tools can be used to adapt to the inevitable emergent requirements.

The idea that a burdensome methodology will improve business systems quality is utter nonesense. The handful of systems that finally makes it through might be of some sort of higher quality by some measure somewhere. But the shanty town of spreadsheet hell that sprouts up to cover those things that didn’t make it in or out really really isn’t an overall step forward.

These methodologies always stink of waterfall to me, and surely that is the most discredited swear word in business system development.

I think these plan, document, approve, analyse, document, approve, design, document, approve…. processes really increase the risk of building the wrong thing too late. To me they are just so much more risky than partnering with the business to jointly develop the required system with trust. And they back load all the risk as the customer gets zero value until close to the end when if the system meets their then current need all well and good. Compare and contrast with iterative development where the customer gets a rough partially working system that address their most critical needs first and then is gradually extended and expanded to meet their developing need. These methodoligies don’t consider this because they were all developed back in the 60′s when the waterfall was the bible and C was just a glint in K&R’s eyes. (remember the 60′s was 50 years ago!!)

So when the the IS department implements their SLA based, documentation heavy, change control hell what actually happens?
Crap spreadsheets spring up absolutely everywhere (thats the least favourite from above for the hard of reading (I didn’t put offshoring on there)). Like the most tenacious of tenacious weeds these instantly become a huge support burden. The support effort takes away from the replacement development effort and the whole thing spirals down out of control like a dying fly that got burnt on a (non eco, non carcegenic old skool filament) bulb. Buzzz..t.

So thats why I hate them – they directly cause a massive deteriation in organisational data quality and cause major paralysis, scores of missed business opportunities, and lots of frustrated business people.

And thats why I love them – those frustrated business people who are busy building castles in the sand with their ratty spreadsheets who have just had their confidence in their in house IS team shattered will be wanting someone to fix their spreadsheet hell at some point. Thats a job for us.

The other reason I love them is even more mercenry. Places I have worked that adopted these heavy methodoligies have a rapid loss of their more dynamic staff. So there are normally a few vacancies there too. And as a contractor you don’t normally have to adhere to their box ticking extravaganza.

So: hate they because they kill organisations love because they provide great work opportunities.

My biggest relief is that I didn’t cough up for that Prince2 course I considered years ago – I couldn’t work in a Prince2 environment so I’m better off with it not on my cv.

Any of you hiding behind an SLA? checking your traffic light (or kimball ball these days, fashion, eh?) progress KPI dashboard against the inevitable landscape printed gantt chart? (what size paper is yours?(dot matrix fanfold??))

How many of you have worked on short term systems whilst the customer waits for the ‘big fix’? (I have, loads)

cheers

Simon

Pareto gave me whiplash

Monday, 1st March, 2010

Hopefully you all know the pareto principle, the 80:20 rule. The idea is that the first 20% of effort gives 80% of the benefit, and a million other combinations all suggesting that there are worthwhile benefits even in just doing part of a job. Its the rallying call of half job harrys and bodgers everywhere, and all under the guise of increased return on investment.

Don’t get me wrong, I think there are many areas where the 80:20 principle is completely valid. But I’m here to tell you skateboarding is not one of them!

This is a reconstruction obviously, I was actually skating the bowl at the local park. A proper skateboarder came up and did a back side kick turn, where you go up the ramp, and as you slow down, lift the front wheels, rotate around the back wheels in the direction your heels are pointing 180 degrees and come back down.

I did at least 80% the same as him, same approach speed, same angle, similar stance, same smile, but I got my weight slightly wrong and went flying – my board nearly decapitated him, and I landed on my arse, hard.

I got sent a spreadsheet recently to review, it was the same thing, on the surface it seemed reasonable, the formulas were a little long but not to bad, the VBA was long but seemed understandable, there were lots of sheets, but the names seemed to make sense, etc etc.

When I came to properly investigate it though, it was incomprehensible, the developer had used 80% reasonable practice, but had really fallen on his arse. I won’t go into the gory detail but the model was a monster.

I’m all for 80/20, maximising return on investment, cutting out features where the cost outweighs the benefit, but I think its important to recognise that you can’t do that everywhere. In most of the work that I do a system that is 80% right is 100% wrong and 100% useless, or worse.

What areas are there that you can think of, where pareto does not apply?

cheers

Simon

And yes, thanks for asking, I can move my head again finally.


Follow

Get every new post delivered to your Inbox.

Join 62 other followers