Object-Relational Model¶
Trump’s persistent object model, made possible by it’s object-relational model (ORM), all starts with
a Symbol, and an associated list of Feed objects.
An fragmented illustration of the ORM is presented in the three figures below.
Supporting objects store details persistently about error handling, sourcing, munging, and validation, so that a Symbol can cache()
the data provided from the various Feed objects,
in a single datatable or serve up a fresh pandas.Series at anytime. A symbol’s it’s Index, can further enhance the intelligence that Trump can serve via pandas.
The full ORM, excludes the symbol’s datatable.
The Symbol portion of the ORM, excludes the symbol’s datatable.
The Feed, FailSafe & Override portion of the ORM
The Index portion of the ORM.
Note
Trump’s template system consists of objects, which are external to the ORM. Templates are used to expedite construction of ORM objects. Nothing about any template, persists in the database. Only instatiated ORM objects would do so. Templates, should be thought of as boilerplate, or macros, to reduce Feed creation time.
Symbol Manager¶
Conversion Manager¶
Symbols¶
Indices¶
A Symbol object’s Index stores the information
required for Trump to cache and serve data with different types of pandas indices.
Warning
A Trump Index does not contain a list of hashable values, like a pandas
index. It should not be confused with the datatable’s index, however it is used in the creation
of the datatable’s index. A more appropriate name for the class might be IndexCreationKwargs.
Index Types¶
Validity Checking¶
Centralized Data Editing¶
Each trump datatable comes with two extra columns beyond the feeds, index and final.
The two columns are populated by any existing Override and FailSafe objects which survive
caching, and modification to feeds.
Any Override will get applied blindly regardless of feeds, while the FailSafe objects are used
only when data isn’t availabe for a specific point. Once a datapoint becomes available for a specific
index in the datatable, the failsafe is ignored.
Reporting¶
During the cache process, information comes back from validity checks, and any exceptions.
This area of Trump’s code base is currently WIP, however the basic idea is that the caching of a
Feed, returns a FeedReport. For each cached Feed, there would
be one report, all of which would get aggregated up into, and combined with the symbol-level information,
in a SymbolReport. When the SymbolManager caches one or more
symbols, it aggregates SymbolReports into one big and final
TrumpReport.
Each of the three levels of reports, have the appropriate aggregated results, plus collections of their own HandlePointReport and ReportPoint objects.
Error Handling¶
The Symbol & Feed objects have a single SymbolHandle and FeedHandle object accessed via their .handle attribute. They both work identically. The only difference is the column names that each have. Each column, aside from symname, represents a checkpoint during caching, which could cause errors external to trump. The integer stored in each column is a serialized BitFlag object, which uses bit-wise logic to save the settings associated with what to do upon an exception. What to do, mainly means deciding between various printing, logging, warning or raising options.
The Symbol’s possible exception-inducing handle-points include:
- caching (of feeds)
- concatenation (of feeds)
- aggregation (of final value column)
- validity_check
The Feed’s possible exception-inducing handle-points include:
- api_failure
- feed_type
- index_type_problem
- index_property_problem
- data_type_problem
- monounique
For example, if a feed source is prone to problems, set the api_failure to print the trace by setting the BitFlag object’s ‘stdout’ flag to True, and the other flags to False. If there’s a problem, Trump will attempt to continue, and hope that there is another feed with good data available. However, if a source should be reliably available, you may want to set the BitFlag object’s ‘raise’ flag to True.
BitFlags¶
Trump stores instructions regarding how to handle exceptions in specific points of the cache process using a serializable object representing a list of boolean values calleda BitFlag. There are two objects which make the BitFlag implementation work. There is the BitFlag object, which converts dictionaries and integers to bitwise logic, and then there is the BitFlagType which give SQLAlchemy the ability to create columns, and handle them appropriately, containing BitFlag objects.
The likely values assigned, will commonly be from the list below. Use Bitwise logic operators to make other combinations.
| Desired Effect | BitFlag Instantiation | Description |
|---|---|---|
| Raise-Only | BitFlag(1) | Raise an Exception |
| Warn-Only | BitFlag(2) | Raise a Warning |
| Email-Only * | BitFlag(4) | Send an E-mail |
| DBLog-Only * | BitFlag(8) | Log to the Database |
| TxtLog-Only | BitFlag(16) | Text Log |
| StdOut-Only | BitFlag(32) | Standard Output Stream |
| Report-Only | BitFlag(64) | Report |
| TxtLog and StdOut | BitFlag(48) | Print & Log |
- Denotes Features not implemented, yet.
The implementation is awkard, all in the name of speed. There are (4 + 7 x # of Feeds) BitFlags, per symbol. So they are serialized into integers, rather than having (4 + 7 x # of Feeds) x 7 boolean database columns.