Object-Relational Model¶

Trump’s persistent object model, made possible by it’s object-relational model (ORM), all starts with a Symbol, and an associated list of Feed objects.

An fragmented illustration of the ORM is presented in the three figures below.

Supporting objects store details persistently about error handling, sourcing, munging, and validation, so that a Symbol can cache() the data provided from the various Feed objects, in a single datatable or serve up a fresh pandas.Series at anytime. A symbol’s it’s Index, can further enhance the intelligence that Trump can serve via pandas.

The full ORM, excludes the symbol’s datatable.

The Symbol portion of the ORM, excludes the symbol’s datatable.

The Feed, FailSafe & Override portion of the ORM

The Index portion of the ORM.

Note

Trump’s template system consists of objects, which are external to the ORM. Templates are used to expedite construction of ORM objects. Nothing about any template, persists in the database. Only instatiated ORM objects would do so. Templates, should be thought of as boilerplate, or macros, to reduce Feed creation time.

Symbol Manager¶

class SymbolManager(engine_or_eng_str=None, loud=False, echo=False)¶

Bases: object

The SymbolManager maintains the SQLAlchemy database session, and provides access to object creation, deletion, searching, and overrides/failsafes.

Parameters:	engine_or_eng_str (str or None, optional) – Pass a SQLAlchemy engine, or a string. Without one, it will use the string provided in trump/options/trump.cfg If it fails to get a value there, an in-memory SQLlite session would be created. loud (bool, optional) – Print information such as engine string used, defaults to False echo (bool, optional) – If a new engine is created, it will pass this to it’safes constructor, enabling SQLAlchemy’s echo mode.
Returns:
Return type:	SymbolManager

add_fail_safe(symbol, ind, val, dt_log=None, user=None, comment=None)¶

Appends a single indexed-value pair, to a symbol object, to be used during the final steps of the aggregation of the datatable.

With default settings FailSafes, get applied with lowest priority.

Parameters:

symbol (Symbol or str) – The Symbol to apply the fail safe
ind (obj) – The index value where the fail safe should be applied
val (obj) – The data value which will be used in the fail safe
dt_log (datetime) – A log entry, for saving when this fail safe was created.
user (str) – A string representing which user made the fail safe
comment (str) – A string to store any notes related to this fail safe.

add_override(symbol, ind, val, dt_log=None, user=None, comment=None)¶

Appends a single indexed-value pair, to a symbol object, to be used during the final steps of the aggregation of the datatable.

With default settings Overrides, get applied with highest priority.

Parameters:

symbol (Symbol or str) – The Symbol to override
ind (obj) – The index value where the override should be applied
val (obj) – The data value which will be used in the override
dt_log (datetime) – A log entry, for saving when this override was created.
user (str) – A string representing which user made the override
comment (str) – A string to store any notes related to this override.

build_view_from_tag(tag)¶

Build a view of group of Symbols based on their tag.

Parameters:	tag (str) – Use ‘%’ to enable SQL’s “LIKE” functionality.

Note

This function is written without SQLAlchemy, so it only tested on Postgres.

complete()¶

Commits any changes to the database. In general, most of Trump API’s auto-commits or does so internally.

This is necessary when working directly with SQLAlchemy exposed attributes.

create(name, description=None, units=None, agg_method='priority_fill', overwrite=False)¶

Create, or get if exists, a Symbol.

Parameters:	name (str) – A symbol’s name is a primary key, used across the Trump ORM. description (str, optional) – An arbitrary string, used to store user information related to the symbol. units (str, optional) – This is a string used to denote the units of the final data Series. agg_method (str, optional) – The aggregation method, used to calculate the final feed. Defaults to priority_fill. overwrite (bool, optional) – Set to True, to force deletion an existing symbol. defaults to False.
Returns:
Return type:	Symbol

delete(symbol)¶

Deletes a Symbol.

Parameters:	symbol (str or Symbol) –

finish()¶

Closes the session with the database.

Call at the end of a trump session. It also calls SessionManager.complete().

get(symbol)¶

Gets a Symbol based on name, which is expected to exist.

Parameters:	symbol (str or Symbol) –
Returns:
Return type:	Symbol
Raises:	`Exception` – If it does not exist. Use .try_to_get(), if the symbol may or may not exist.

search(usrqry=None, name=False, desc=False, tags=False, meta=False, stronly=False, dolikelogic=True)¶

Get a list of Symbols by searching a combination of a Symbol’s name, description, tags or meta values.

Parameters:	usrqry (str) – The string used to query. Appending ‘%’ will use SQL’s “LIKE” functionality. name (bool, optional, default False) – Search by symbol name. desc (bool, optional, default False) – Search by symbol descriptions. tags (bool, optional, default False) – Search by symbol tags. meta (bool, optional, default False) – Search within a symbol’s meta attribute’s value. stronly (bool, optional, default True) – Return only a list of symbol names, as opposed to the (entire) Symbol objects. dolikelogic – Append ‘%’ to either side of the string, if the string doesn’t already have % specified.
Returns:
Return type:	List of Symbols or empty list

search_meta(attr, value=None, stronly=False)¶

Get a list of Symbols by searching a specific meta attribute, and optionally the value.

Parameters:	attr (str) – The meta attribute to query. value (None, str or list) – The meta attribute to query. If you pass a float, or an int, it’ll be converted to a string, prior to searching. stronly (bool, optional, default True) – Return only a list of symbol names, as opposed to the (entire) Symbol objects.
Returns:
Return type:	List of Symbols or empty list

search_meta_specific(**avargs)¶

Search list of Symbol objects by by querying specific meta attributes and their respective values.

Parameters:	avargs – The attributes and values passed as key word arguments. If more than one criteria is specified, AND logic is applied. Appending ‘%’ to values will use SQL’s “LIKE” functionality.

Example

>>> sm.search_meta(geography='Canada', sector='Gov%')

Returns:
Return type:	List of Symbols or empty list

search_tag(tag, symbols=True, feeds=False)¶

Get a list of Symbols by searching a tag or partial tag.

Parameters:	tag (str) – The tag to search. Appending ‘%’ will use SQL’s “LIKE” functionality. symbols (bool, optional) – Search for Symbol’s based on their tags. feeds (bool, optional) – Search for Symbol’s based on their Feeds’ tags.
Returns:
Return type:	List of Symbols or empty list

Conversion Manager¶

class ConversionManager(engine_or_eng_str=None, system='FX', tag=None)¶

Bases: trump.orm.SymbolManager

A ConversionManager handles the conversion of previously instantiated symbols, based on the object’s units and the conversion manager setup. The conversion is performed adhoc, in python only usage. That is, nothing about the conversion persists in the Trump framework. Only the final series is converted.

Parameters:

engine_or_eng_str (str or None) – Pass a SQLAlchemy engine, or a string. Without one, it will use the defaul provided in trump/options/trump.cfg If it fails to get a value there, an in-memory SQLlite session would be created.
system (str, optional) –
Uses the FX conversion system logic by default. Currently, no other systems are implemented. Eg. metric-only, imperial-metric, etc.

Other systems can be added after instantiation of the ConversionManager, but the one specified at instantiation will be used as default.
tag (str, optional) –
Tag for the set of feeds to use for conversion. Only necessary, if the conversion system relies on it. For FX, it’s needed, to specify the set of feeds to use.

Other tags can be added after instantiation of the ConversionManager, but the one specified at instantiation will be used as default.

get_converted(symbol, units='CAD', system=None, tag=None)¶

Uses a Symbol’s Dataframe, to build a new Dataframe, with the data converted to the new units

Parameters:

symbol (str or tuple of the form (Dataframe, str)) – String representing a symbol’s name, or a dataframe with the data required to be converted. If supplying a dataframe, units must be passed.
units (str, optional) – Specify the units to convert the symbol to, default to CAD
system (str, optional) – If None, the default system specified at instantiation is used. System defines which conversion approach to take.
tag (str, optional) – Tags define which set of conversion data is used. If None, the default tag specified at instantiation is used.

Symbols¶

class Symbol(name, description=None, units=None, agg_method='PRIORITY_FILL', indexname='UNNAMED', indeximp='DatetimeIndexImp', freshthresh=0)¶

Bases: sqlalchemy.ext.declarative.api.Base, trump.tools.sqla.ReprMixin

A Trump Symbol persistently objectifies indexed data

Use the SymbolManager class to create or retrieve existing symbols.

Parameters:

name (str) – The name of the symbol to be added to the database, serves as a primary key across the trump installation.
description (str, optional) – a description of the symbol, just for notes.
units (str, optional) – a string representing the units for the data.
agg_method (str, default PRIORITY_FILL) – the method used for aggregating feeds, see trump.aggregation.symbol_aggs.py for the list of available options.
indexname (str) – a proprietary name assigned to the index.
indeximp (str) – a string representing an index implementer (one of the classes in indexing.py)
freshthresh (int, default 0) – number of minutes before the feed is considered stale.

add_feed(feedlike, **kwargs)¶

Add a feed to the Symbol

Parameters:	feedlike (Feed or bFeed-like) – The feed template, or Feed object to be added. kwargs – Munging instructions

add_tags(tags)¶

add a tag or tags to a symbol

Parameters:	tags (str or [str,]) – Tags to be added

add_validator(val_template)¶

Creates and adds a SymbolValidity object to the Symbol.

Parameters:	validity_template (bValidity or bValidity-like) – a validity template.

cache(checkvalidity=True, staleonly=False, allowraise=True)¶

Re-caches the Symbol’s datatable by querying each Feed.

Parameters:	checkvalidity (bool, optional) – Optionally, check validity post-cache. Improve speed by turning to False. staleonly (bool, default False) – Set to True, for speed up, by looking at staleness allowraise (bool, default True) – AND with the Symbol.handle and Feed.handle’s ‘raise’, set to False, to do a list of symbols. Note, this won’t silence bugs in Trump, eg. unhandled edge cases. So, those still need to be handled by the application.
Returns:
Return type:	SymbolReport

check_validity(checks=None, report=True)¶

Runs a Symbol’s validity checks.

Parameters:	checks (str, [str,], optional) – Only run certain checks. report (bool, optional) – If set to False, the method will return only the result of the check checks (True/False). Set to True, to have a SymbolReport returned as well.
Returns:
Return type:	Bool, or a Tuple of the form (Bool, SymbolReport)

describe¶: describes a Symbol, returns a string

isvalid¶

Quick access to the results of a a check_validity report

Returns:
Return type:	Bool

set_indexing(index_template)¶

Update a symbol’s indexing strategy

Parameters:	index_template (bIndex or bIndex-like) – An index template used to overwrite all details about the symbol’s current index.

to_json()¶: Returns the json representation of a Symbol object’s tags, description, and meta data

update_handle(chkpnt_settings)¶

Update a symbol’s handle checkpoint settings

Parameters:

chkpnt_settings (dict) –

a dictionary where the keys are stings representing individual handle checkpoint names, for a Symbol (eg. caching_of_feeds, feed_aggregation_problem, ...) See SymbolHandle.__table__.columns for the current list.

The values can be either integer or BitFlags.

class SymbolTag(tag, sym=None)¶: Bases: sqlalchemy.ext.declarative.api.Base, trump.tools.sqla.ReprMixin

class SymbolDataDef(datadef, sym=None)¶: Bases: sqlalchemy.ext.declarative.api.Base, trump.tools.sqla.ReprMixin

Indices¶

A Symbol object’s Index stores the information required for Trump to cache and serve data with different types of pandas indices.

Warning

A Trump Index does not contain a list of hashable values, like a pandas index. It should not be confused with the datatable’s index, however it is used in the creation of the datatable’s index. A more appropriate name for the class might be IndexCreationKwargs.

class Index(name, indimp, case=None, kwargs=None, sym=None)¶

Bases: sqlalchemy.ext.declarative.api.Base, trump.tools.sqla.ReprMixin

case¶: string used in a IndexImplementer switch statement.

indimp¶: string representing a IndexImplementer.

name¶: string to name the index, only used when serving.

class IndexKwarg(kword, val)¶: Bases: sqlalchemy.ext.declarative.api.Base, trump.tools.sqla.ReprMixin, trump.tools.sqla.DuckTypeMixin

Index Types¶

class IndexImplementer(case, **kwargs)¶

Bases: object

IndexImplementer is the base required to implement an index of a specific type. The same instance is created at two points in the Trump dataflow:

the datatable getting cached and
the data being served.

The IndexImplementer should be indempotent, and dataframe/series agnostic.

Parameters:	case – str This should match a case used to switch the logic created in each subclass of IndexImplementer kwargs – dict

pytyp¶: alias of int

sqlatyp¶: alias of Integer

class DatetimeIndexImp(case, **kwargs)¶

Bases: trump.indexing.IndexImplementer

Implements a pandas DatetimeIndex

Cases include:

asis - Cache timestamps to the database and drop any intelligence associated with the index, such as frequency. serve a Series with a DatetimeIndex, without frequency.

If the index consists of 4-digit integers, it will be treated as the year, in a date which is of the form YYYY-12-31.
asfreq - Apply ‘asfreq’ logic prior to cache, and apply the same logic when serving.
date_range - Create a new index, using pandas date_range(), at time of cache... NotImplemented yet.
guess - NotImplemented yet. Attempt to guess the frequency at time of cache, and time of serve.
guess_post - NotImplemented yet. Attempt to guess the frequency at time of serve, but store the cache unsaved.

In the event that case hasn’t implemented the logic to handle a specific datatype, a rudimentary attempt to convert it to a DatetimeIndex is applied by inspecting the start and end, with the kwargs. passed pandas.DatetimeIndex constructor.

Parameters:	case – str This should match a case used to switch the logic created in each subclass of IndexImplementer kwargs – dict

pytyp¶: alias of datetime

sqlatyp¶: alias of DateTime

class PeriodIndexImp(case, **kwargs)¶

Bases: trump.indexing.IndexImplementer

Implements a pandas PeriodIndex

NotImplemented, yet.

sqlatyp¶: alias of DateTime

class StrIndexImp(case, **kwargs)¶

Bases: trump.indexing.IndexImplementer

Implements a pandas Index consisting of string objects.

Only method, is “asis”

Parameters:	case – str This should match a case used to switch the logic created in each subclass of IndexImplementer kwargs – dict

pytyp¶: alias of str

sqlatyp¶: alias of String

class IntIndexImp(case, **kwargs)¶

Bases: trump.indexing.IndexImplementer

Implements a pandas Int64Index.

Cases include:

asis - attempts to pass the index through, without applying any logic. Use this, if the index is already integers, or unique and integer-like.
drop - will drop the pandas index, to reset it.

Parameters:	case – str This should match a case used to switch the logic created in each subclass of IndexImplementer kwargs – dict

pytyp¶: alias of int

sqlatyp¶: alias of Integer

Validity Checking¶

class SymbolValidity(symbol, vid, validator, *args)¶: Bases: sqlalchemy.ext.declarative.api.Base, trump.tools.sqla.ReprMixin

Feeds¶

class Feed(symbol, ftype, sourcing, munging=None, meta=None, fnum=None)¶

Bases: sqlalchemy.ext.declarative.api.Base, trump.tools.sqla.ReprMixin

The Feed object stores parameters associated with souring and munging a single series.

class FeedMeta(feed, attr, value)¶: Bases: sqlalchemy.ext.declarative.api.Base, trump.tools.sqla.ReprMixin

class FeedSource(stype, sourcing_key, feed)¶: Bases: sqlalchemy.ext.declarative.api.Base, trump.tools.sqla.ReprMixin

class FeedSourceKwarg(kword, val, feedsource)¶: Bases: sqlalchemy.ext.declarative.api.Base, trump.tools.sqla.ReprMixin, trump.tools.sqla.DuckTypeMixin

Feed Munging¶

class FeedMunge(order, mtype, method, feed)¶: Bases: sqlalchemy.ext.declarative.api.Base, trump.tools.sqla.ReprMixin

class FeedMungeKwarg(kword, val, feedmunge)¶: Bases: sqlalchemy.ext.declarative.api.Base, trump.tools.sqla.ReprMixin, trump.tools.sqla.DuckTypeMixin

Centralized Data Editing¶

Each trump datatable comes with two extra columns beyond the feeds, index and final.

The two columns are populated by any existing Override and FailSafe objects which survive caching, and modification to feeds.

Any Override will get applied blindly regardless of feeds, while the FailSafe objects are used only when data isn’t availabe for a specific point. Once a datapoint becomes available for a specific index in the datatable, the failsafe is ignored.

class Override(*args, **kwargs)¶

Bases: sqlalchemy.ext.declarative.api.Base, trump.tools.sqla.ReprMixin

An Override represents a single datapoint with an associated index value, applied to a Symbol’s datatable after sourcing all the data, and will be applied after any aggregation logic

comment¶: a user field to store an arbitrary string about the override

dt_log¶: datetime that the override was created

ind¶: the repr of the object used in the Symbol’s index.

ornum¶: Override number, uniquely assigned to every override

symname¶: symbol name, for the override

user¶: user name or process name that created the override

val¶: the repr of the object used as the Symbol’s value.

class FailSafe(*args, **kwargs)¶

Bases: sqlalchemy.ext.declarative.api.Base, trump.tools.sqla.ReprMixin

A FailSafe represents a single datapoint with an associated index value, applied to a Symbol’s datatable after sourcing all the data, and will be applied after any aggregation logic, only where no other datapoint exists. It’s a back-up datapoint, used only by Trump, when an NA exists.

Note

only datetime based indices with float-based data currently work with Overrides

comment¶: user field to store an arbitrary string about the FailSafe

dt_log¶: datetime of the FailSafe creation.

fsnum¶: Failsafe number, uniquely assigned to every FailSafe

ind¶: the repr of the object used in the Symbol’s index.

symname¶: symbol name, for the override

user¶: user name or process name that created the FailSafe

val¶: the repr of the object used as the Symbol’s value.

Reporting¶

During the cache process, information comes back from validity checks, and any exceptions. This area of Trump’s code base is currently WIP, however the basic idea is that the caching of a Feed, returns a FeedReport. For each cached Feed, there would be one report, all of which would get aggregated up into, and combined with the symbol-level information, in a SymbolReport. When the SymbolManager caches one or more symbols, it aggregates SymbolReports into one big and final TrumpReport.

class FeedReport(num)¶

Bases: object

add_handlepoint(hpreport)¶: Appends a HandlePointReport

add_reportpoint(rpoint)¶: Appends a ReportPoint

asodict(handlepoints=True, reportpoints=True)¶: Returns an ordered dictionary of handle/report points

class SymbolReport(name)¶

Bases: object

add_feedreport(freport)¶: Appends a FeedReport

add_handlepoint(hpreport)¶: Appends a HandlePointReport

add_reportpoint(rpoint)¶: Appends a ReportPoint

asodict(freports=True, handlepoints=True, reportpoints=True)¶: Returns an ordered dictionary of feed, and handle/report points

class TrumpReport(name)¶: Bases: object

Each of the three levels of reports, have the appropriate aggregated results, plus collections of their own HandlePointReport and ReportPoint objects.

class HandlePointReport(handlepoint, trace)¶: Bases: object

class ReportPoint(reportpoint, attribute, value, extended=None)¶: Bases: object

Error Handling¶

The Symbol & Feed objects have a single SymbolHandle and FeedHandle object accessed via their .handle attribute. They both work identically. The only difference is the column names that each have. Each column, aside from symname, represents a checkpoint during caching, which could cause errors external to trump. The integer stored in each column is a serialized BitFlag object, which uses bit-wise logic to save the settings associated with what to do upon an exception. What to do, mainly means deciding between various printing, logging, warning or raising options.

The Symbol’s possible exception-inducing handle-points include:

caching (of feeds)
concatenation (of feeds)
aggregation (of final value column)
validity_check

class SymbolHandle(chkpnt_settings=None, sym=None)¶

Bases: sqlalchemy.ext.declarative.api.Base, trump.tools.sqla.ReprMixin

Stores instructions about how to handle exceptions thrown during specific points of Symbol caching:

sh = SymbolHandle({'aggregation' : BitFlag(36)}, aSymbol)
>>> sh.aggregation['email']
True

Parameters:	chkpnt_settings (dict) – A dictionary with keys matching names of the handle points and the values either integers or BitFlags sym (str or Symbol) – The Symbol that this SymbolHandle is associated with it.

The Feed’s possible exception-inducing handle-points include:

api_failure
feed_type
index_type_problem
index_property_problem
data_type_problem
monounique

class FeedHandle(chkpnt_settings=None, feed=None)¶

Bases: sqlalchemy.ext.declarative.api.Base, trump.tools.sqla.ReprMixin

Stores instructions about specific handle points during Feed caching:

fh = FeedHandle({'api_failure' : BitFlag(36)}, aSymbol.feeds[0])
>>> fh.api_failure['email']
True

Parameters:	chkpnt_settings – dict A dictionary with keys matchin names of the handle points and the values either integers or BitFlags feed – Feed The feed that this FeedHandle is associated with it.

For example, if a feed source is prone to problems, set the api_failure to print the trace by setting the BitFlag object’s ‘stdout’ flag to True, and the other flags to False. If there’s a problem, Trump will attempt to continue, and hope that there is another feed with good data available. However, if a source should be reliably available, you may want to set the BitFlag object’s ‘raise’ flag to True.

BitFlags¶

Trump stores instructions regarding how to handle exceptions in specific points of the cache process using a serializable object representing a list of boolean values calleda BitFlag. There are two objects which make the BitFlag implementation work. There is the BitFlag object, which converts dictionaries and integers to bitwise logic, and then there is the BitFlagType which give SQLAlchemy the ability to create columns, and handle them appropriately, containing BitFlag objects.

class BitFlag(obj, defaultflags=None)¶

Bases: sqlalchemy.ext.mutable.Mutable, object

An object used to encode and decode a boolean array as an an integer representing bitwise logic-flags.

There are 7 hardcoded flags:

raise
warn
email
dblog
txtlog
stdout
report

Each can be set to True or False, with convenience either at instantiation, or key-base set operations.

Example of instatiation, setting the email and stdout flag to True:

BitFlag(['email','stdout'])

Example of instatiation, setting the email then, later setting stdout flag to True:

bf = BitFlag(['email'])
bf['stdout'] = True

After either running either of these, the BitFlag will have a value of:

>>> bf.val == 36
True

>>> print bf
raise warn EMAIL dblog txtlog STDOUT report

>>> print bf.bin_str
00100100

>>> print bf.email
True

...because the 3rd and 6th bit are set.

Warning

Flag state can be read from the accessors named after the flags, however, they can’t be written to.

Parameters:	(int, dict) (obj,) – either the decimal form of the bitwise array, or a dictionary (complete or otherwise) of the form {flag : bool, flag : bool, ...} dict (defaultflags,) – a dictionary representing the default for one or more flags. Only applicable when a dictionary is passed to obj. It’s ignored when obj is an integer.

__and__(other)¶

Parameters:	other – int, BitFlag

BitFlag and integers work with the and operator using bitwise logic.

Returns:	BitFlag

__or__(other)¶

Parameters:	other – int, BitFlag

BitFlag and integers work with the or operator using bitwise logic.

Returns:	BitFlag

asdict()¶: convert the flags to a dictionary, with keys as flags.

bin¶: the binary equivalent

bin_str¶: the binary equivalent, as a string

class BitFlagType(*args, **kwargs)¶

Bases: sqlalchemy.sql.type_api.TypeDecorator

SQLAlchemy type definition for the BitFlag implementation. A BitFlag is a python object that wraps bitwise logic for hardcoded flags into a single integer value for quick database access and use.

The likely values assigned, will commonly be from the list below. Use Bitwise logic operators to make other combinations.

Desired Effect	BitFlag Instantiation	Description
Raise-Only	BitFlag(1)	Raise an Exception
Warn-Only	BitFlag(2)	Raise a Warning
Email-Only *	BitFlag(4)	Send an E-mail
DBLog-Only *	BitFlag(8)	Log to the Database
TxtLog-Only	BitFlag(16)	Text Log
StdOut-Only	BitFlag(32)	Standard Output Stream
Report-Only	BitFlag(64)	Report
TxtLog and StdOut	BitFlag(48)	Print & Log

Denotes Features not implemented, yet.

The implementation is awkard, all in the name of speed. There are (4 + 7 x # of Feeds) BitFlags, per symbol. So they are serialized into integers, rather than having (4 + 7 x # of Feeds) x 7 boolean database columns.