|
Ensign Database Design, New Bar Styles, TWS Upgrade |
October 10 , 2007
Welcome to today's training class where I will be introducing and discussing new features in a forthcoming version. I have invited a couple of users to download what I am calling the Alpha version. I ONLY want these couple of users to try the program and provide feedback. Once these initial users are running smoothly, then I will invite a few more on different data feeds. After we have happy users for a week, we will release for a general Beta version where, if interested, you can invite yourself to be a beta tester. Do not invite yourself to be an Alpha version tester, because I will not support anyone other than the few I have invited already.
| Bar Styles | Database Design | Auto Refresh | Playback | TWS Upgrade |
I will show today many of the great features that hopefully are just a week away from a more general release. The Ensign Windows program has undergone the most major rewrite in its 15 year life span. I started on these major rewrite back in late July, and several of the initial changes were released in August as a beta version. Many users have been beta testers, and are using the latest beta release dated Sept 9th. See the What's New page for a list of the Beta features already released, such as the rework of the price histogram study, and changes to property forms.
Most of the existing features have been reviewed in previous classes and will not be repeated today.
Lets begin with the 8 new bar styles that are available in the alpha version.
The new styles are on the drop down list for the Bar Styles. The first is the Ensign Z-Flute, which is a variation of the Ensign Flute to add a Z line from the Open to the Close, like the Ensign Zebra does.
The C-Flutes shows just the Close, and does so as a block in the middle of the pipe.
ticks: WoW Howard this is already impressive and brilliant ! thanks
We also had a request for a flute without any open and close marking, so that is the X-Flute, were X means the information is omitted.
The Line and Close style is like the Line chart and the Close Ticks chart together, and you can make the line thickness bolder with the line thickness selection.
bld: I have already created line and close charts with template. Use them all time. They are great!
Great bld, now with the new selections you will be able to speed up your chart by having the program do the visual instead of having a template do it.
The Line and Range selection adds the bar's range to the Line and Close style chart.
The Line and Band selection shows the Line and Close chart with a band created by connecting the Highs and the Lows.
@Buffy2: Band style can't be changed right?
Correct.
The Bar Fat Close selection implements the request to have a fatter marker for the close tick on a OHLC bar chart.
The Bar High/Low selection shows a bar with lines at the high and low, and the close is the bullet on the pole. The open is not shown on this style.
I think Gann might have liked to use this last selection. OK, that introduces the 8 new bar styles that you will enjoy when this version is released to the general public.
The really major change to the program has been the redesign of the foundation database. In the past, we have had chart files that were updated by live ticks if the chart was open, and brought current by extracting ticks from a tick pool if the chart was closed. The chart would be brought current from the tick pool when it was opened. This worked, but had the following limitations which we wanted to overcome.
The tick pool had a capacity for 3 million ticks, which might be a few days of capacity depending on the number of symbols followed and their activity level. If a chart based on ticks, such as V1000, was not opened for 2 weeks, the tick pool would not be able to bring it current. The tick pool would not have 2 weeks of ticks. Therefore, the chart would show bars previously stored, then a gap, then the last few days of ticks from the tick pool. Other charts based on the same ticks might be current because they had been viewed to be updated before the needed ticks dropped out of the tick pool.
Another limitation could be the tick pool becomes corrupt because of a computer freeze which required the computer to be rebooted. Needed pointers from our quote table to the entry point in the tick pool might not have been saved, and thus the tick pool becomes useless, needs to be purged and the collection started over again. The impact of some of these limitations was minimized by having tick refresh available from the data vendors.
Another limitation about the refresh is that if you refreshed ticks for the V1000 chart that was open, this refresh did not benefit any of the closed charts, such as the R200 or the V500 charts. So you would end up refreshing again when those close charts were opened.
The database redesign improves on all of the above issues. This was accomplished in the following manner.
We have a new Database Manager form accessed from menu File | File Manager | Database Manager, or from a chart's pop-up menu. On the left side you will have a folder for Ticks\ and a folder for Minutes\. In each folder are sub-folders containing the database for each symbol that you follow. So there is a tick database and a separate 1-minute database. Each day is a separate file providing the following advantages.
1) If you need to replace or repair a day in the database, do so by working on that day's file. The whole database is never at risk of being corrupted or lost.
When you select a date in the database, a row of boxes show that days' contents. There is a box for each half hour of the day and the colors indicate something about the content in that half hour period. The 24 hour day will thus have 48 boxes, beginning with the evening session open and going through the day session close.
A box that is dark blue has data obtained by refreshing. A box that is green has data obtained from the live data feed...... and a refresh of the same period will change green to blue. A box in the aqua color has no data, but that was the way it was in the refresh, so the omission is probably correct and doing another refresh is not going to add anything, because you already had a refresh covering that period. Most likely, the market is just idle or closed during that time period.
A box that is white indicates that no data has been seen (yet) in real-time or in a refresh, such as the remainder of the current day.
@Buffy2: half of it 4:15 to 4:30?
In your example Buffy, the box would get colored because of data from 4:00-4:15. A red box, indicates that data is missing in the past, and may just needs to be refreshed. Now, in the example, on the right there are boxes that have a center hole.... this indicates those time periods have not been locked.
If the data is correct, and you do not want it to be changed, click the Lock
File button and the boxes with holes become solid boxes. A refresh
will make the box solid to begin with, meaning it is already as good at it
can get and refreshing again will not benefit. The green box
could be made locked by locking it, in which case the refresh will not replace
it.
To unlock a file that was locked, there is an Unlock File button, in which
case a refresh could then replace the bars. This lock/unlock system will
replace the Preserve Existing bars which was a private chart property.
@Buffy2: but work on tick and volume now?
Yes, since this lock/unlock feature works on the tick database files, it is another advantage over the old tick pool system which did not offer any want to preserve bars.
Now the database is a blend as follows. When a chart is opened, it will load the existing chart file, IE your ES #F.V1000 chart which might go way back. Then the database for ticks is used to bring it current. The database can have many months of files, and these do not self delete. Thus, in the earlier example where a chart had not been opened for 2 weeks, now is not a problem. These 2 weeks of tick files exist so the chart can be updated. Then the new updated chart file is resaved so the extraction is not needed the next time it is opened.
The building of bars from the new database is super fast, much faster than extracting from the tick pool. And any updating done to the database by either the live feed or the refresh benefits all charts that are closed. A single refresh that adds to the database provides content for any chart that uses the database. The same principles apply to the minute database.
Abinadi: Hooray
ticks: Bravo
@Buffy2: backup your data!
All intra-day charts read from the same 1-min bar files to be brought current, and the process is super fast. So you will notice a speed increase as you open workspaces, etc. Also, you will have much less need to do any refreshing. In the past, if you opened a 5-min chart, 30-min, 60-min, 13-min, etc, each had its own refresh request. Now, at most, would be a single refresh request for 1-min data that then benefits all these open charts and all close charts yet to be opened.
price: wonderful
ticks: Brilliant Howard -- thanks
dom993: Howard, does this leaves second-charts out?
@Buffy2: what about volume charts?
Not at all dom993, seconds charts build from the tick database. Tick, range, seconds, and volume charts all build from the tick database. Now, this leads us to the next cool feature, and that is to have the program be smarter about what needs to be refreshed.
On the Setup | Charts property form are 2 check boxes to turn on what I call an automatic refresh feature that kicks in when the program runs. When these boxes are checked and Ensign runs, then the program will go through the symbols on the Setup | Manager list and determine how much data is needed to bring the databases current, and make refresh requests automatically.
bld: A wonderful program for 25 years gets better and better! Refreshing data was always somewhat difficult. This seems to answer that problem.
Thus, if the feature is enabled, you might have an updated database within a couple minutes of running Ensign and thus not see another refresh request all day long. Or, you can have the auto refresh on program run turned off, and then when the chart is opened, we see it might not have a current database, and go do a refresh and show the data when it is received. Either way works well.
Abinadi: If we leave Ensign running all the time, should we restart it each day now?
No, Abinadi. If on-line, you have all the data. You can still right mouse click and ask for a refresh, or you can go to the database manager, and click the refresh button on that form to refresh that symbol. Both places allow you to pick the amount to download.
Now this is really cool.... and exciting.
If you select the DTN Refresh as the source, you can refresh 1 year of 1-min files in a single request. The refresh of 12 months of 1-min files takes less than 2 minutes, and would only be needed to be done once for a symbol. This gives you lots of back data for all the intra-day charts, and from then on you are just keeping the database updated each day via the live feed or an occasional refresh.
Continuation Files:
Nick: is your data base going to correct for DTN's early roll date compared to other providers?
No Nick, it does not change DTN's roll date. If you are using eSignal, then refresh from eSignal to have the roll date for continuation files they way they do it. That would be the Data Vendor bullet selection. Also, on the DTN roll date topic, you can do this... collect the data live, such as March data, through the roll day. Lock the file so you do not accidentally replace any of the march data by refreshing with June data. That way the file integrity should be preserved that it is march data on a particular day.
Also, you can select a day, and change the cross reference symbol to be a particular symbol, such as march, and refresh for 1 or more days to put march data in the database. Then change the calendar date, and the cross reference, and refresh to put June data in the database on other days. So you will have more control than before, particularly by locking data once it is the way you want it.
Nick: but if I want a chart on a new future symbol I am stuck with bad data in the days before the roll date
Nick, I assume you are wanting the eSignal roll schedule data.... so another solution is to have your eSignal friend e-mail you particular date files for your database. You would replace individual file dates, not the whole database. Then you will match your eSignal friend though you may have initially built your continuation files using DTN refresh.
At the top of the Database Manager form are some statistics for the day, such as the open, high, low and close for each session and for the day. We also show the Up Volume and Down Volume and number of ticks, etc.
@Buffy2: Howard does it matter if I refresh with right click or on the dbm window?
terry: nice. find the data and throw the files in.
Buffy, either way is the same.
Existing Files:
Kongfuzi: is it possible to use/plot OLD historic files on S&P and ES years old?
Yes, kongfuzi, old files load and display same as before. They probably will not find data in the database to be brought current, so they will show as opened/loaded from the old historic file. Any refresh will update both the database and any open chart. Any closed chart will benefit from the previous refresh because it is brought current from the database. The former tick pool has been obsoleted and removed.
Time & Sales:
Time and Sales now is a view into the tick database, so you can quickly jump to any date and any time in that date, and scroll up or down. And you can view the tick database as a tick chart by clicking the Chart button on the time and sales form. T&S can be shown from the chart, from a quote page, or by clicking the Time & Sales button on the new Database Manager form.
Here is an earlier example, but the principles shown still apply.
The database redesign also impacted the Playback feature. In the past, we had special compressed playback files used for the playback. That has been replaced with playing back the actual ticks from the tick database. So what you collect or refresh is the source file for the playback. There is no need to do anything more, and there is no longer a Build Playback Files process.
@Buffy2: so no more playing back anything we don't collect?
Buffy,you can playback what you have, and we can still post playback files for download like we have done in the past. The current implementation still plays back a single symbol into the DEMO symbol, but another enhancement I want to work on is an ability to playback multiple symbols simultaneously. The playback files will now be the tick database files for a symbol for a day. So the playback download could be an alternate way to get the database files from our web site instead of by refreshing from your data vendor. Our files would continue to be built from the eSignal feed/refresh.
Our IB customers will like all these improvements because refresh for them has been a challenge. All these changes make for less refresh, which will be of benefit to them. We are also working to make the program smarter to handle or avoid the IB pacing violation requirement, which limits the number of refresh requests to 60 in a 10-min period.
The revisions made since mid July have been the biggest undertaking in the history of Ensign Windows. We have more work yet to do, and hence the alpha phase, which will be followed by a beta phase of the database.
ticks: Howard what can we say but this is really great and Viva Ensign, Viva Howard ! Bravo !
@Buffy2: very nice for sure Howard
Charles3: thanks Howard - excellent work
Fortunately, the current beta phase has many of the improvements well under control. Once I get past the database changes, I can return to working on implementing other good ideas, of which there are many. But the big database redesign needs to be finished and polished off first.
ticks: Why isn't every trader using Ensign ??
MakesOwnWeather: Ensign Howard, I have three question categories:
Q: I do not yet understand about how past data
will be handled when moving to the new system. It seems from what you
say that the old data be "sucked" into the new database (a big plus IMO). Correct? Additionally,
will any updating of the data be made? For instance, if I have some range files
with 65k bars and some volume / tick bars with 65k bars but many with far lesser
bars, will the new database be able to fill in the range files but not the
tick / volume files? Or, will the range depend on the granularity of
the files that I now have?
A: Old files continue in their current location.....
database replaces tick pool and not the data files. So old files
are brought current from database instead of tick pool, and then resave as updated
file. So no change really, in your legacy files. You
can have some big files and some smaller files. Those file sizes are still
controlled by the Maximum Bars chart property. This is independent of the
size of the foundation tick database. You will be able to increase
Maximum Bars to a larger value, and rebuild older bars from the tick database
so you have more bars in the chart.
Q: One of the challenges which I have with the current data structure in Ensign Softeare is I must test and research with one time frame (eg ER2 2800) but I can not trade with that time frame as with the many, many bars of the data files (that one of others which I save 65000 bars), I must trade with the 2801V or the 2799V; otherwise, I will run out of memory and also system will be VERY slow (and I know you are working on speed).
Would there be a way in the new system to keep the 65000 bars on the 2800V that I trade with and have a limited number of bars in the chart while trading without truncating the file. (I suppose since you are building a DB, that the answer would be Yes.)
A: Answer will be Yes. The database will keep the
data for the ticks so you can have a chart with a small size without erasing
the foundation database. Then if you what a test file, make the Max
Bar count big again and rebuild the chart to restore the bars way back. Database
will have the ticks regardless of what you do with the Max Bars setting for
the chart, which is just a view of a portion of the master database.
Q: Also, I have found that the DTN refresh rounded
off minutes (on tick / volume data) so the charts were slightly (although significant
to my research) different from live data which did not round the minutes. Will
this change to non-rounding on refresh?
A: I am unaware of any rounding..... the DTN refresh
can be different than the live collection, and that fact may have nothing to
do with rounding. I have never seen agreement with live feeds from DTN,
quote.com, eSignal or IB. Each is different, and since worrying does
not change that fact, I ignore the issue and just move on. It
is only you folks comparing yourself to someone else that you notice there
is a difference. Difference exists, and bringing it to my
attention repeatedly is not going to resolve anything.
ticks: Ain't that the truth
The difference exists at the source. I just gather data from the feed and show the content... we are the bucket at the end of the pipeline. We can catch and show, but we cannot invent or control what is put into the pipeline.
MakesOwnWeather: Well, yes, but not exactly, Ensign Howard. I am comparing my IB feed with DTN and DTN refresh with IB. (IB more accurately reflects actual price transactions than DTN refresh)
Q: What about standard symbols? For instance,
will there be any integration between the DTN symbols and the IB symbols? (Like
replacing / eliminating the silly @ with a # or coming to some standardization
for the most commonly used symbols.)? It seems to me that Ensign Software is
in control of the symbol assignment and is the one place were a standardization
could best work for Ensign Software users; we have several data feeds and it
would be good for us if there was standardization.
A: Standardization has been implemented more than
you realize.... for example we have ES #F as a symbol for you IB users to use. That
is not an IB symbol, and for those on something like a DTN feed or you want
to use the DTN refresh, we do have automatic cross reference symbols in the program. So
you can use your IB symbol and get DTN refresh for it.
We have to use cross reference symbols for each source. For example, take
the Dow Jones index. On eSignal it is $INDU, and we use the same symbol
and our behind the scene magic to make it work as the symbol for the IB feed. The
DTN symbol would be INDU.X, and the Yahoo symbol would be ^DJI
Q: Is it possible to limit the number of days in the data manager or to just disable it?
A: I gather you are talking about the Database Manager demonstrated today. It is needed to bring charts current, so it cannot be disabled. However, you do not have to build lots of back files via a refresh as illustrated today. And if you abandon a symbol, there is a Delete Folder button on the form to remove all files and the folder that is selected. Each day is a file, so you can delete individual back files if you want to remove them. But hard disks are getting cheap. Each file is around 56K in size.
Possibly we can add to the Chart | Maintenance form a task to remove database files that are older than x days, so the program could be used to delete a group of older files. Now you can use Window Explorer to navigate to the folders, list the files by date, and delete a group of files. This illustrates another advantage of having individual files by date, instead of all data buried in one big file or database.
In summary, I am getting close to being done with the huge project. I am in alpha testing with 2 invited users and will invite more as time passes and the first users are taken care of. I hope to be able to move to a general beta phase in 1 week.
ticks: Howard what can we say but this is really great and Viva Ensign, Viva Howard !
One final discussion point is about the TWS upgrades. Back in march starting with TWS version 873.3 and then with 875.3, IB changed the way the bars were timestamped in the refresh when using either of those released, you needed to check a box on the Setup | Connection form to tell Ensign to adjust for the IB change.
Then a week ago, starting with TWS 877.5, then switched back to the old time stamp method. So with TWS 877.5, 877.7 and 877.8 you need to have the box unchecked. We have had a bulletin announcement on the top of our web site and Wiki site about this need to uncheck the TWS box on the Setup | Connection form.
In the alpha version, I have made the decision to not support older TWS or older IQFeed versions. So the TWS check box has been removed, and the expectation is you will be upgraded to the latest version of the vendor software.