Comments on: ETL tools

By: Jim

Jim — Wed, 13 May 2009 15:56:51 +0000

HI Simon, since you used access access for data cleaning process can you link me with some notes how access works as an ETL tool? I know access is Not an Etl tool but it has some characteristics of an ETl tool and thats what I am inquiring about…

By: Runrig

Runrig — Tue, 24 Mar 2009 20:29:22 +0000

My opinions about ETL tools can summed up and linked from here: http://stackoverflow.com/questions/51198/what-etl-tool-do-you-use/67784#67784

I once had a talk over lunch with a manager about how, for our department’s needs, custom programming/scripts would be simpler, faster to develop, more maintainable, cheaper, etc. Every reason he had for not using custom scripts came from a position of not understanding of what was possible with them (you can still connect to every database, XML source, flat files, even spreadsheets), and his points were easily refuted. Except for one: that with an ETL tool, the developer is more replaceable.

By: charlie murphy

charlie murphy — Mon, 30 Jun 2008 16:12:49 +0000

I just finished a project using the pervasive etl tools and was pretty pleased. I have not used any others etl tools but I could write an entire book about how this was the best thing I've ever discovered. Just not having to custom code saved me days of work on that one project. Going forward I will try another brand, not becasue anything was wrong with pervasive just that I want to try somebody else too.

By: Will Riley

Will Riley — Thu, 21 Jun 2007 20:27:06 +0000

Well we use SSIS (the replacement for DTS in SQL 2005)

I have no issues with it. As a non-programmer, I find it does everything I require in an extremely efficient & intuitive manner. Most of our source systems are Oracle, yet the BI stack is a mixture of SQL 2000 & SQL 2005 – SSIS is the most efficient tool we tested for getting data from the source systems to the SQL BI Data Warehouse….

By: Simon

Simon — Wed, 13 Jun 2007 17:04:24 +0000

Thanks Harlan, I had a feeling you would know the command.
Marcus, Informatica is one that is being considered, glad to hear there may be some ‘opportunities’ there!
cheers
Simon

By: Marcus

Marcus — Tue, 12 Jun 2007 22:05:31 +0000

I love the smell of irony. I’m in the middle of a migration project at the moment, moving a Monte Carlo simulation model from Excel/Access to a production environment as the technology utilised does not meet the bank’s standards. For ETL the project is using Informatica. For the Informatica developer I wrote a VB6 DLL to extract and cleanse the data received in Excel workbooks as Informatica (apparently) wasn’t up to the task.
Negatives:
It took the business about 3 months to get a suitable Informatica resource (they’re a bit light the ground at the moment).
Data Types – it seems to see everything a strings.

I try to use Access and queries (SQL) for ETL, otherwise I’ll use Excel or VB6. I have worked on a project where the IT department coded the ETL functionality in C++. While it took a while to code, performance was critical. For some other “production environment” projects, we’ve use DTS.

By: Ross

Ross — Tue, 12 Jun 2007 09:49:30 +0000

Ha!

Simon, I was doing this type of thing (in VB.Net though) just yesterday.
To start off with i thought, .net must have a classs to do all this stuff, then i googled for it and after a bit of looking around found this:

http://filehelpers.sourceforge.net/

Which I tried, then i thought, bugger that, I just do it with bog standard code – i read what the chap ad to say, but for me i could really see the point.

I have some code now that runs though (puts in array) a 8000 x 2 CSV in about 0.0000001 seconds – not tried it with bigger file – but i think it will be fine.

By: Harlan Grove

Harlan Grove — Tue, 12 Jun 2007 03:27:44 +0000

General tool? One could claim C is as general a tool as C#. Then there are the standard POSIX tools. For your example,

sed “s/a/0/g” inputfile > outputfile

I’ve been using Unix-like tools so long that I just couldn’t adapt to overly specialized ETL tools. That said, does Monarch count as an ETL tool?