I made a mistake and bought UltraFS. It was highly recommended by the AAII Los Angeles Computerized Investing Special Interest Group. It was a frustrating experience. Hopefully the following will save money, time, effort, or even erroneously relying on a bad tool.
This is a Back-Testing tool. I will gloss over the sever limitations of Back-Testing, like tricky underlying assumptions or how to exploit its results. I am focusing on the quality of the UltraFS tool and my experience with the tool vendor.
This tool has extremely limited functionality, implemented by faulty code, covers limited set of trading systems, uses incorrect modeling of the trading systems, some of the underlying theoretical basis for the implemented systems are either erroneous or missing key components, documentation is meager and misleading, support is non-existence, the simplistic database (end of day) has errors and contains only small subset of equities and indices, the produced results are unsatisfactory, …. you get the idea.
I did spend over two weeks (three full weekends) to study the tool. Actually big chunk of time was used to create a post-process tool based on Visual Basic and Excel. This additional effort was needed to generate some rudimentary missing results (e.g. volatility) or badly missing functionality (e.g. graphics).
Some of the specific UltraFS issues include non-deterministic executions (could not repeat results using identical parameters), multiple and varied implementations of identical functions (got different results based on which part of the code was invoked), discrepancies between results based on either implicit (and default) parameters, explicit parameters, or hand calculated, erroneous stored database values or pulling wrong values from the database, faulty invocations of aggregation of systems (called Composites by UltraFS), … you get the idea.
Many issues were recorded with detailed information. Perhaps at this stage I can also add some guessing about the reasons for the poor quality of the product. For instance it was probably created long time ago, with a DOS environment, as evident by the 8 character limitation or by math that definitely does not crack the 16 bit barrier. Or the multiple implementations of a single function suggest poor software practices, or big time-gaps between implementation updates, or perhaps both.
I had documented multiple issues and sent them to UltraFS. A couple of years ago I managed to receive one single response from Steve Hunter (chief developer) which was long and incoherent (mostly consisted of “cut and paste” from other sources). Again, it was send over two years ago. It basically stated the following:
--Users need to learn the “nuances” of the tools to produce correct results.
--All issues will be cleared by the next version (#11) which is around the corner.
--All issues are quickly addressed and recorded on the UltraFS website.
--Users should monitor the UltraFS.com website for complete and accurate responses to open issues.
Needless to say that monitoring the UltraFS.com site for many months, yielded nothing. I did receive a couple of emails (over two years ago) from Melinda Brown (new administrator at that time), but they were irrelevant to the issues I raised.
Over a year ago I did presented my finding to the AAII Los Angeles Computerized Investing Special Interest Group as part of venting my frustration. You can find the slides on the Investment Education Meeting - AAII-LA. During my presentation (in front of over 45 people) I executed UltraFS live and demonstrated a plethora of serious execution errors and faulty results. I also used one simple single system (picked arbitrarily by others) for seven execution cycles. The first pass was used for to establish a baseline. The other six passes produced various error types. Those execution errors were verified independently by the leader of that AAII group.
Several persons reported erroneous results following my initial email reports, and later, after my presentation to the AAII group. Still, the public image and the widely published messages contain only praise and adulation to the UltraFS software. Hopefully I can spur a useful discussion with inputs from the UltraFS vendor and experienced users. At least I hope to warn the uninitiated from wasting time and money.
Yes, I got a chip on my shoulder. I am trying to get my money back for a very long time. But UltraFS does not answer the phone, does not return voice messages, and does not answer any of my emails. I am writing this message six weeks after my last cycle of attempts (ended on January 23rd). This last cycle was trigger by UltraFS request for additional money though I had never been a user of their software and stopped monitoring their website for suggested resolution, as I informed them long time ago.
Please note that my early communication attempts with UltraFS used constructive approach. I did offer to become an UltraFS tester, I did propose additional functionality that is easy to implement, I did sent my post-process Excel file to UltraFS, I did suggest ways to utilize UltraFS (see my presentation slides), and so on. Of course I received no response.
Ed
This is a Back-Testing tool. I will gloss over the sever limitations of Back-Testing, like tricky underlying assumptions or how to exploit its results. I am focusing on the quality of the UltraFS tool and my experience with the tool vendor.
This tool has extremely limited functionality, implemented by faulty code, covers limited set of trading systems, uses incorrect modeling of the trading systems, some of the underlying theoretical basis for the implemented systems are either erroneous or missing key components, documentation is meager and misleading, support is non-existence, the simplistic database (end of day) has errors and contains only small subset of equities and indices, the produced results are unsatisfactory, …. you get the idea.
I did spend over two weeks (three full weekends) to study the tool. Actually big chunk of time was used to create a post-process tool based on Visual Basic and Excel. This additional effort was needed to generate some rudimentary missing results (e.g. volatility) or badly missing functionality (e.g. graphics).
Some of the specific UltraFS issues include non-deterministic executions (could not repeat results using identical parameters), multiple and varied implementations of identical functions (got different results based on which part of the code was invoked), discrepancies between results based on either implicit (and default) parameters, explicit parameters, or hand calculated, erroneous stored database values or pulling wrong values from the database, faulty invocations of aggregation of systems (called Composites by UltraFS), … you get the idea.
Many issues were recorded with detailed information. Perhaps at this stage I can also add some guessing about the reasons for the poor quality of the product. For instance it was probably created long time ago, with a DOS environment, as evident by the 8 character limitation or by math that definitely does not crack the 16 bit barrier. Or the multiple implementations of a single function suggest poor software practices, or big time-gaps between implementation updates, or perhaps both.
I had documented multiple issues and sent them to UltraFS. A couple of years ago I managed to receive one single response from Steve Hunter (chief developer) which was long and incoherent (mostly consisted of “cut and paste” from other sources). Again, it was send over two years ago. It basically stated the following:
--Users need to learn the “nuances” of the tools to produce correct results.
--All issues will be cleared by the next version (#11) which is around the corner.
--All issues are quickly addressed and recorded on the UltraFS website.
--Users should monitor the UltraFS.com website for complete and accurate responses to open issues.
Needless to say that monitoring the UltraFS.com site for many months, yielded nothing. I did receive a couple of emails (over two years ago) from Melinda Brown (new administrator at that time), but they were irrelevant to the issues I raised.
Over a year ago I did presented my finding to the AAII Los Angeles Computerized Investing Special Interest Group as part of venting my frustration. You can find the slides on the Investment Education Meeting - AAII-LA. During my presentation (in front of over 45 people) I executed UltraFS live and demonstrated a plethora of serious execution errors and faulty results. I also used one simple single system (picked arbitrarily by others) for seven execution cycles. The first pass was used for to establish a baseline. The other six passes produced various error types. Those execution errors were verified independently by the leader of that AAII group.
Several persons reported erroneous results following my initial email reports, and later, after my presentation to the AAII group. Still, the public image and the widely published messages contain only praise and adulation to the UltraFS software. Hopefully I can spur a useful discussion with inputs from the UltraFS vendor and experienced users. At least I hope to warn the uninitiated from wasting time and money.
Yes, I got a chip on my shoulder. I am trying to get my money back for a very long time. But UltraFS does not answer the phone, does not return voice messages, and does not answer any of my emails. I am writing this message six weeks after my last cycle of attempts (ended on January 23rd). This last cycle was trigger by UltraFS request for additional money though I had never been a user of their software and stopped monitoring their website for suggested resolution, as I informed them long time ago.
Please note that my early communication attempts with UltraFS used constructive approach. I did offer to become an UltraFS tester, I did propose additional functionality that is easy to implement, I did sent my post-process Excel file to UltraFS, I did suggest ways to utilize UltraFS (see my presentation slides), and so on. Of course I received no response.
Ed
