|
Class: 11-13-07 Playback and Database |
Welcome to today's training class and the primary topic will be the Playback feature, which many who upgraded to the new Ensign have questions about. This was also a topic for discussion in yesterday's training class, but will be repeated here to some degree. We will also do a playback exercise so those who participate will feel more comfortable with playback as a result of the training. Playback Training: The old Ensign used specially built playback files to play back a prior date's ticks. The new Ensign has the perfect resource already available in the form of the tick database files. So the new Ensign will use the tick database files now instead of the special playback files. That is the essence of the change. So first of all, lets go through the step to see what you have for a tick database. The tick files must exist in order to use them in a playback exercise. Lets use for the exercise the ES #F symbol which is valid for IB, eSignal and Open Tick feeds. Those on the IQFeed would use @ES#, and those on TransAct would use =ESZ7. Click menu File | File Manager | Database Manager and on the form have the Ticks bullet selected. On the left side list select the ES #F or equivalent folder.
The left side is a directory list with a companion column of file names in the selected folder. My example shows I have tick files for ES #F symbol for every day for the past couple of weeks. I have ES #F and NQ #F tick database files on the Ensign web site that can be downloaded using the Internet Services form. The download process if you elect to get my web files will put the files in the right tick folders. So, IQFeed and TransAct users can go get the ES #F files for 071108 and 071109 which we will use in today's exercise. The files I have for download on my web site were built with refresh data from eSignal. If you build files from DTN the tick counts will be slightly different.
Another step you can do if you not have a file or your file is incomplete, is to select the date using the calendar tool on this form, select a quantity of Today or 1 Day, select your source and click the Refresh button. If your vendor does not have the data, you can change the source to the DTN Refresh selection and get the data from DTN. So the first step I am showing is seeing what data is available and helping you get data if you need to.
Go ahead and close the database manager form, and click menu Setup | Playback to show the playback form.
Enter in the Symbol field the symbol you will do a playback for. Today we will use ES #F as the symbol. You can either select the date using the calendar tool on the form, OR you can browse to the tick database folder by clicking the browse button.
If you browse to the folder, select a file on the list and click the open button. We will use Friday's date of 11-09 for the playback. The Filenames are in the format of yymmdd which is 071109 for Nov 9th, 2007.
You would select a speed, such as the 4x which is 4 times normal speed. A 1 minute period would play in 15 seconds using 4x.
The Start Time is the point at which you want to start the playback in the day. Often this is the open time such as 9:30 Eastern time. For the example, lets use the 1035 entry which then is an hour into the day session. The time can be entered as either 10:35 or as 1035. Either format is tolerated.
The Sync time would be blank if you are doing the playback by yourself. If you are trying to be coordinated and in sync with others in a chat room doing a playback exercise, then a mentor needs to start a playback, and the Sync time will fill in automatically. This Sync time would then be told to the others in the room, and they too will then be in sync with the others in the room by entering the sync time provided by the mentor. Let me start a playback and be the one to provide the rest of you the Sync time to enter.
Shar: Do we use EST if our computer is different zone? Like PST?
All times on the playback form are to be entered using Eastern zone time. After entering the sync time, such as the 16:56:42 to be in sync with me, then you click the Open Demo Chart button.
tim: What is the "Play Back Enabled" box for and do we need to have it checked ?
@Buffy2: I use reset and then just open demo workspace
Now I typically click the Open Demo Chart button and that will open a chart for the time frame listed by the button, which in the example is 1 for a 1-min chart. You can do the following instead. You can check the Play Back Enabled box, which starts the playback, and click OK to close the form. Then you might open a workspace containing DEMO symbols, or use the charts panel to open a DEMO chart. So either way works, and I personally use the Open Demo Chart button because it does all of the following: checks Enable, closes form, opens chart. So that was a 3 in 1 click.
This chart shows where the playback has progressed so since I started a few minutes ago.
@Buffy2: I have been asked why 4x and 6x is tradeable with new version and sure wasn't on old Ensign as it went too fast -- is it due to the new steady speed in new version?
That is correct.
Data ahead of the start time will be showing already on the chart, and this can include multiple prior days if the tick files for those prior days exist in the database folder. That is why I suggested that you also have 11-08 tick file for ES #F in the folder. The charts build the bars from the tick database very fast, much faster than the old Ensign. You can put the same templates, draw tools, and studies on the playback charts as you use on your real charts. The ONLY difference about the playback is the faster speed such as 4x, and the fact that only a single symbol feed is being played.
tim: The template name in my chart properties window shows a template name from today but the template on the Demo chart is actually from about 2 months ago so the template name is incorrect in the chart properties window.
Tim, it shows what was on the chart when you last closed it. So go put on the new template. That is not a playback question but would apply to any chart you open.... it show how it was last dressed, which can be from 2 months ago too
Saroj: is there any way to download a group of tick data or is it only one day / one instrument at a time? (for playback)
Saroj, one file at a time, or refresh one file at a time. The files to download are around 1.5 Meg per file, so it is 1 at a time.
@Buffy2: love the fact we seem to be staying in scyc with new way.
Max: What is the purpose of the playback feature? Is it for paper trading? If for back testing, can you show an example pls?
Playback is for training...... some will paper trade, some will practice recognition of the signals and setups they are looking for. Some will will research and verify their systems and signals trigger in real-time.
JohnA: how do you shut it off?
To stop the playback return to the Playback form and uncheck the Enable Playback checkbox.
Saroj: ahh... so no more Pause at the top?
@Buffy2: you can use the pause and resume button if you want a break also. But you will not stay in sync with anyone using this.
Good point Buffy, if you pause the others doing the same playback went on. If the mentor pauses, he should reset the Start Time, clear the Sync box, start himself, and post a new playback form for others to get the new start and sync times.
Saroj: I have found that if I didn't blank out the sync time in the earlier version, it would start at an odd time... not the start time... is that still the case?
Yes. Still the same.
@Buffy2: if you are starting one alone than yes the sync time should be blank. @Buffy2: gee ezgoing if I didn't remember to blank it mine wouldn't run in old version -- you are lucky.
ezgoing: This is doing great Howard, I had had problems using my range charts with the old playback but no prob now
tim: Sarjo, I have noticed if I load a large amount of data on a chart i.e. 20,000 bars, it will cause Ensign to lock up during real trading ( your Q about prior tick files )
Large amount of data is not a problem.
daveM: I had what I thought was a 'lock-up', but it was just the machine processing a huge new file. took a couple extra secs to get happening
Database Training:
Now let me give some training on the topic of of using the database manager to have lots of back data. I will use ES #F 15-min as an example. This is not related to Playback, but is training about using the database manager.
Here I have the property from for the 15-min chart with the Max Bars set to a big size of 62000 bars. On the chart I will right mouse click and select pop-up menu Database Manager. I changed the Source to DTN Refresh, the quantity to 1 year, the Select box to the Continuation bullet so we will use the DTN continuation symbol instead of the Dec 2007 symbol.
When I click the Refresh button it will go get 1 year of 1 minute data from DTN. It can take a few seconds before the refresh starts to respond, and several seconds to receive. The counter on the top of the form will count up to 197000 bars and then populate the database with a year's worth of files. Now you can change the calendar date to the first file on the top of the list and refresh again another year. There is data for ES for this far back. In doing the refresh twice I have populated database files all the way back to Oct 2005.
Then on the 15-min chart, which initially did not have data that far back, I did the following. I hold down the shift key while pressing = key, and this jumps the chart to the first bar in the existing chart. Then right mouse click and use pop-up menu Rebuild Visible. The chart sees the 1st bar is visible, and realizes you want to try to get data ahead of that bar, and it will populate the chart will all the data downloaded, and the chart then goes back to Oct 2005. The build of all this data is very fast, and the chart repaints then with the 13661 bars I showed on the chart properties data tab. This is how I got such a big file for ES #F 15-min.
The chart was day session only, and all I have to do is change the state of that to include all sessions. There is a keyboard hot key to toggle day session only on or off and that is the @ character, (Shift 2 key).
This is the chart with days session only, all downloaded from DTN using the database manager as discussed above. I pressed shift 2 (@ key) and in less than 2 seconds the chart with 47320 bars showed, all newly built from the database. The chart did not exist before I pressed the key to go to all session.
33
Now let me jump to beginning of chart with shift = keys and repost.
Here is the chart back in Nov 2005. As I scroll through the chart for the past 2 years, I AM VERY PLEASED and IMPRESSED with how clean the data is for this download from the DTN database. I do not see spikes.
Now, consider what we have just done. For a symbol of interest, I have just downloaded a database for 2 years (ES had that much). It is continuation data, and now that it is downloaded on the computer I will not need to do that ever again. All intra-day charts will benefit from this nice database. If you want a new 119 minute chart for what ever reason.... presto you have it. It did not require any new download or refresh. It builds the bars from the existing database, and the tiny refreshes of polish off a recent day's collection is about all the refresh you might be doing. So that is all exciting.
@Buffy2: it is exciting for sure Howard once all understand it.
IB Refresh:
@Buffy2: OK so an IB user turns on machine in morning - charts need updating -- they just right click and refresh? or go to dbm and refresh?
For IB user, I would recommend the open a chart, and right mouse click and refresh it as needed.
@Buffy2: so they are back on one refresh for time charts and one for R V T right?
Too many IB are having challenges with the auto refresh on program run so I discourage IB users from using that. Also, IB users are encouraged to refresh from DTN when possible because that source is both fast and does not count against the pacing violation imposed by IB. But Buffy, a single refresh received by anyone will benefit all time frames for that symbol.
@Buffy2: if IB users refresh a Tick chart that is good for range and volume also
Yes, and that is a huge benefit.
Saroj: so Howard - IB users should always have "DTN Market Access" checked in the Data UI of Chart Properties?
Not always saroj. The following needs to be considered by IB users. I used the word encouraged to use DTN because DTN does not have the same symbol universe as IB. IB might be the only source for refreshing the symbol. Also, DTN is through a delay requirement, and IB can be through current tick. Also, DTN will have more ticks in the refresh of tick databases then IB will have sent in real time, and that bothers some to have tick or volume charts changed by refreshing. So, there is more to consider than just the fact that DTN is fast, has more days back, etc.
On the tick based chart, when an IB user right mouse clicks they have on the pop-up menu refresh selections that would get data from IB and one for getting data from DTN. So they can choose which to use on the pop-up menu regardless of the setting on the data tab of chart properties. And on the database form there is the source bullet selection which would control the source for the refresh.
Saroj: I'm still not clear on how to decide when to use which... and the impact (if any) on how the chart properties "refresh source" is set up on intra-day refresh. I just downloaded the new version an hour ago... but I'm not clear about when to use IB, when to use DTN, and the interrelationship of the chart properties to the data manager UI
We prepare a refresh request structure and one of the properties is the source, since there might be more than one source with the data, such as from the data vendor or from DTN market access. That selection is a bullet choice on the database manager form. The same selection is also on the chart property form on the data tab if that is your starting point. The selection is also on the pop-up menu for IB tick chart refreshers.
So where ever you are working from, you make the choice, the request is made and data flows into files in the database folders. If you click Refresh on the database form, then that source selection is used independent of any chart that is open. Likewise, if you are refreshing form a chart, then that chart's properties are used for the source. It is all the same, makes no difference. The database form is just a way to see what files exist and make a refresh request. The chart is just a 2nd way to make a refresh request. They both call the same procedure to do the refresh, and the reply is identical regardless of whether a chart is open or the database manager is open.
Saroj: so to sum up... if both IB and DTN have the same symbol set, use DTN for all but very current data (like the last hour); otherwise use IB?
That is a good rule of thumb
Saroj: ok... that helps a lot... a nice, simple "rule of thumb"... thanks. And it seems to me that I should leave the Data tab set to IB so when I do a quick refresh intra-day (in case I had to reboot or lost power for instance) it will get the current data from IB?
Saroj: Well, Howard, this is a very welcome change... having this database of data right on ones machine... solves all kinds of problems... thank you for implementing this.
@Buffy2: agree saroj just need to get used to it
Welcome, and I agree it has lots of benefits.
@Buffy2: works well. I didn't know it had the problems which you are having. But then I use eSignal feed.
Shar: Thanks for you time in coming in and helping. Sure is making things clearer.
@Buffy2: yes thank you it is getting clearer and clearer :-))