dqpy

Danqing’s shared Python library, written with love for the better Danqing.

Cache

The cache module provides a simple JSON and Redis based caching decorator.

dq.cache.cache(ttl=600, key_func=None)

Cache decorator.

This can be applied to any function that returns a raw or JSON-serializable response. To allow caching, the cache key must be set in the config, namely the redis connection for cache.

If the function has a keyword argument named fresh, then the decorator gets a fresh copy when it’s set to a truthy value.

If the function has a keyword argument named raw, then the decorator returns the raw (bytes) Redis response as-is, without JSON-deserializing.

Parameters:

• ttl (number) – The TTL in second. Default is 10 minutes.
• key_func (func) – The key function. This function should take the same arguments as the wrapped function, and return the corresponding cache key as a string.

Config

The config module helps getting configurations from TOML files stored in the config/ folder. The default file to use is config/local.toml. To use another file, specify the DQENV environment variable. For example, if DQENV=production, then the config/production.toml file will be used. Nested keys are comma-separated.

class dq.config.Config

Bases: object

Configuration helper.

classmethod get(key)

Get the config value specified by the provided key.

TOML config files should be stored in the config folder, and the environment (i.e. which file to choose) should be specified by the DQENV environment variable.

Parameters:key (string) – The key to retrieve the value for. Nested key should have its components separated by dot, such as owner.email. Returns:The value of the key if it exists, and None otherwise.

classmethod init()

Initialize the configuration.

If the config file contains a dictionary keyed by env, all its content will be set as environment variables.

This function should not be called externally. The Config.get function lazily initializes the config object automatically.

Database

SQL database client. This module provides connectors to the SQL database, with its configs specified (by you) in the config files. Check out the config/local.toml file for an example configuration.

dq.database.commit_scope(session=None)

Commit any database operations within this scope.

dq.database.query_with_limit_offset(query, limit, offset)

Add limit and offset constraints to a SQLAlchemy query.

Parameters:

• query (Query) – A SQLAlchemy query.
• limit (int) – The limit to add. If None or 0, it will be skipped.
• offset (int) – The offset to add. If None or 0, it will be skipped.

Returns Query:

The updated query object.

dq.database.replace_insert(insert, compiler, **kw)

Allow replace into for insert command.

This only works for MySQL. It’s not standard SQL. To enable this function, the sql.flavor config must be set to mysql.

dq.database.safe_save_to_database(model, session=None)

Safely write things to the database.

This function simply calls unsafe_save_to_database under the hood, but returns whether the operation is successful instead of throwing an error upon failure.

Parameters:

• model (object/list) – The model object(s) to save. If a list is provided, the elements will be added one by one. Otherwise the object is added as-is.
• session (Session) – The optional database session to use.

Returns boolean:  

True if the operation is successful.

dq.database.save_to_database(model, session=None)

Write things to the database.

This function simply calls unsafe_save_to_database under the hood, but wraps the error into a ModelError upon failure.

Parameters:

• model (object/list) – The model object(s) to save. If a list is provided, the elements will be added one by one. Otherwise the object is added as-is.
• session (Session) – The optional database session to use.

Raises:

ModelError – if the save fails.

dq.database.session_maker()

Generate a scoped session. In general, use the Session global variable.

Returns Session:  A new SQLAlchemy session.

dq.database.unsafe_save_to_database(model, session=None)

Convenient method for writing things to the database.

If the object is a list, its elements will be written to the database one by one. If the object is already in the database (i.e. when we are updating existing records), the caller of this function is responsible of ensuring that the same session used when retrieving the records is passed in. If a different session is provided, this operation will fail. (This is usually not an issue for view functions, which have session consistency on a thread level.)

Parameters:

• model (object/list) – The model object(s) to save. If a list is provided, the elements will be added one by one. Otherwise the object is added as-is.
• session (Session) – The optional database session to use.

DBAdmin

This module is designed to be used as a shell script, for creating and dropping the database.

dbadmin create  # Create a database with URL at sql.url
dbadmin drop    # Drop the above database
dbadmin create another_sql.url  # Create a database with URL at anothersql.url

Email

Module for sending emails.

dq.email.send_email(to, subject, html, sender_name, sender_email)

Send an email.

Parameters:

• to (string) – The recepient of the email.
• subject (string) – The subject of the email.
• html (string) – The HTML content of the email.
• sender_name (string) – The name of the sender.
• sender_email (string) – The sender address of the email.

Returns boolean:  

True if the email is sent successfully. False otherwise.

Entity

The entity module provide basic functionalities for Schematics entity classes.

class dq.entity.Entity(raw_data=None, trusted_data=None, deserialize_mapping=None, init=True, partial=True, strict=True, validate=False, app_data=None, lazy=False, **kwargs)

Bases: schematics.deprecated.Model

Base entity class that provides shared utilities.

Currently it only allows printing a schematics entity. More utilities may be added in the future. In any case, all entities should inherit from this class.

Errors

The errors module contains a bunch of error classes so we don’t need to redefine them over and over. They also translate to HTTP errors nicely.

exception dq.errors.DQError(message='', status_code=None, error_code=None)

Bases: Exception

Base exception.

Clients that wish to catch all exceptions should catch this one. The status_code will be set as the HTTP status code if a view handler triggers an error. If there is need to set the exact type of error, use error_code.

4xx errors are due to the user, and as such the message will be passed directly back to user. 5xx errors will have the message replaced by the generic ‘Internal Error’ before sending back to user.

to_dict()

exception dq.errors.ForbiddenError(message='Forbidden')

Bases: dq.errors.DQError

Exception raised when the user does not have the required permission.

exception dq.errors.IntegrationError(message)

Bases: dq.errors.DQError

Exception raised when there is a service integration error.

exception dq.errors.InternalError(message)

Bases: dq.errors.DQError

Exception raised when there is an internal server error.

exception dq.errors.InvalidInputError(message='Invalid input')

Bases: dq.errors.DQError

Exception raised when the user input is not valid.

exception dq.errors.ModelError(message)

Bases: dq.errors.DQError

Exception raised when something is wrong with the models.

exception dq.errors.NotFoundError(message='Not found')

Bases: dq.errors.DQError

Exception raised when something is not found.

exception dq.errors.RecoverableError(message)

Bases: dq.errors.DQError

Exception raised when a recoverable error occurs.

This may be due to network error, timeout, etc. and is meant to be retried. If retrying gives no luck, this is considered a 500 error.

exception dq.errors.UnauthorizedError(message='Unauthorized')

Bases: dq.errors.DQError

Exception raised when the user is not authenticated for an action.

FSUtil

The fsutil module contains file and file system helpers.

dq.fsutil.fileinfo(path)

Information of a file.

This function throws an error if the path is not a valid file. To suppress errors, use safe_fileinfo instead.

Parameters:path (string) – The path to the file. Returns string mime:  The MIME type of the file, such as text/plain. Returns int size:  The size (in bytes) of the file. Returns string encoding:  The encoding of the file, like gzip. This is suitable for use as the Content-Encoding header.

dq.fsutil.mkdirp(path)

Safely mkdir, creating all parent folders if they don’t yet exist.

This doesn’t raise an error if the folder already exists. However, it does raise FileExistsError if the path points to an existing file.

Parameters:path (string) – The path to the folder to be created.

dq.fsutil.rmrf(path)

Remove a path like rm -rf.

Parameters:path (string) – The path to remove.

dq.fsutil.safe_fileinfo(path)

Retrive information of a file without throwing an error.

If the path is invalid, None, 0, None will be returned.

Parameters:path (string) – The path to the file. Returns string mime:  The MIME type of the file, such as text/plain. Returns int size:  The size (in bytes) of the file. Returns string encoding:  The encoding of the file, like gzip. This is suitable for use as the Content-Encoding header.

dq.fsutil.traverse(path, callback)

Traverse a directory recursively, performing a task.

Parameters:

• path (string) – The path of the directory.
• callback (func) – A callback applied to each child file and directory. It takes 2 arguments, the relative path from the root, and whether the item is a directory.

FuncUtil

The funcutil module contains function helpers.

dq.funcutil.safe_invoke(noreturn=False)

Invoke a function safely.

Parameters:noreturn (boolean) – Whether the function being called has no return value itself. Default to False. If True and the function returns successfully, True will be returned. None will be returned upon failure no matter whether the function has return value. Returns func:The decorator function that provides safe invocation.

Logging

The logging module helps format logging messages. It prints the message and pretty-prints the attached data dictionary. Use it as follows:

import logging
from dq.logging import error

logger = logging.getLogger(__name__)
error(logger, 'An error!', {'key': 'value'})

dq.logging.debug(logger, message, payload=None)

Log a debug message. This does nothing in production enviornment.

Parameters:

• logger (logger) – The logger to use.
• message (string) – The debug message.
• payload (dict) – The payload to attach to the log.

dq.logging.error(logger, message, payload=None)

Log an error message.

Parameters:

• logger (logger) – The logger to use.
• message (string) – The error message.
• payload (dict) – The payload to attach to the log.

dq.logging.info(logger, message, payload=None)

Log an info message.

Parameters:

• logger (logger) – The logger to use.
• message (string) – The info message.
• payload (dict) – The payload to attach to the log.

dq.logging.warning(logger, message, payload=None)

Log a warning message.

Parameters:

• logger (logger) – The logger to use.
• message (string) – The warning message.
• payload (dict) – The payload to attach to the log.

ORM

The SQL ORM base classes. Your model classes should most likely inherit from IDBase or UUIDBase, and some other mixins.

class dq.orm.Attr(name, allow_from, allow_to, serializer, deserializer)

Bases: tuple

allow_from

Alias for field number 1

allow_to

Alias for field number 2

deserializer

Alias for field number 4

name

Alias for field number 0

serializer

Alias for field number 3

class dq.orm.DictMixin

Bases: object

Dictionary mixin that provides converters to/from Python dicts.

See comments of Attr above for defining fields to convert with this mixin.

This class is inherited by IDBase and UUIDBase classes, so your class will not need to inherit directly from it.

classmethod from_dict(data)

Convert a dictionary to an object.

Parameters:data (dict) – The data dictionary. Returns Base:The object converted from the dictionary.

inflate_to_dict(session=None)

Convert an object to dictionary, with relationships inflated.

This method is the default implementation, which simply calls the to_dict method. Subclasses willing to inflate certain fields should override this method.

Parameters:session (Session) – The optional SQL session. Returns dict:The dictionary converted from the object.

to_dict()

Convert an object to dictionary.

Returns dict:The dictionary converted from the object.

to_json()

Convert an object to JSON string.

Returns string:The JSON string representation of the object.

class dq.orm.QueryMixin

Bases: object

Query helper functions useful to all models.

This class is inherited by IDBase and UUIDBase classes, so your class will not need to inherit directly from it.

classmethod get_by(column, value, for_update=False, contains_deleted=False, contains_empty=False, session=None)

Get the object satisfying the query condition.

Parameters:

• column (string) – The name of the column to query by.
• value (string) – The value of the column to query for.
• for_update (boolean) – Whether the query is for updating the row.
• contains_deleted (boolean) – Whether to contain deleted records. Default is False and only active records are returned.
• contains_empty (boolean) – Whether to contain empty records. Default is False and if value is None, None will be returned.
• session (Session) – Optional SQL session.

Returns QueryMixin:  

The matching object. This method is designed for unique queries and in case of multiple matches, only the first one is returned.

classmethod get_multi(column, value, sort_column='updated_at', desc=True, offset=0, limit=10, session=None)

Get multiple objects satisfying the query condition.

Parameters:

• column (string) – The name of the column to query by.
• value (string) – The value of the column to query for.
• sort_column (string) – The column to sort results against. Default is updated_at. Please note that created_at usually doesn’t have index, so if it should be used as the sort column, an index needs to be added.
• desc (bool) – Whether to sort DESC. Default is True.
• offset (int) – The query offset. Default is 0.
• limit (int) – The max number of results to return. Default is 10. If specified, the function will try to fetch one more result to indicate whether there are more results. If not specified or 0, all results will be returned.
• session (Session) – The optional SQL session to use.

class dq.orm.TimeMixin

Bases: object

Time fields common to all models.

This class is inherited by IDBase and UUIDBase classes, so your class will not need to inherit directly from it.

created_at = Column(None, ArrowType(), table=None, nullable=False, default=ColumnDefault(<function utcnow>))

deleted_at = Column(None, ArrowType(), table=None)

updated_at = Column(None, ArrowType(), table=None, nullable=False, onupdate=ColumnDefault(<function utcnow>), default=ColumnDefault(<function utcnow>))

class dq.orm.UserRelationMixin

Bases: dq.orm.QueryMixin

Mixin that provides a common get_by_user function.

This mixin should be used by classes that a user can have a 1-many relationship to, such as orders, shipments, and periods.

classmethod get_by_user(user_id, desc=True, offset=0, limit=10, session=None)

Get the related object of a user.

Parameters:

• user_id (string/int) – The ID (if int) or UUID (if string) of the user.
• desc (bool) – Whether to sort DESC. Default is True.
• offset (integer) – The offset of the objects. If limit is 10 and offset is 5, the function will attempt to get the 6th to 15th most recent objects.
• limit (integer) – The max number of objects to get.
• session (Session) – The optional SQL session.

Returns list<UserRelationMixin>:  

The list of objects matching the requirement.

sort_key = None

dq.orm.arrow_in(value)

Convert an arrow-understandable time to an Arrow object.

Parameters:value – The arrow-understandable time or None. Returns Arrow:The Arrow object, or None if the input is None.

dq.orm.arrow_out(value)

Convert an Arrow object to timestamp.

Parameters:value (Arrow) – The Arrow time or None. Returns int:The timestamp, or None if the input is None.

dq.orm.boolean_mask(value)

Mask a value to boolean. This can be used for sensitive fields.

Parameters:value – Any value. Returns boolean:  The input value casted to boolean.

dq.orm.enum_value(value)

Convert an enum to its value.

Parameters:value (Enum) – An enum. Returns string:The value of the enum.

dq.orm.uuid4_string()

Return a UUID as string.

The UUID type must NOT be used anywhere in the app - only use string. This function should be used whenever a new UUID is needed.

Returns string:A UUID string.

Redis

The Redis client. It reads the config files for Redis configs, which you must specify to use this client. Refer to the config/local.toml file in this repo for an example.

class dq.redis.Redis

Bases: object

classmethod atomic_rw(key, evaluator=<function Redis.<lambda>>)

Atomically read-write a Redis key.

Parameters:

• key (string) – The key to read/write.
• evaluator (function) – The evaluator function. It takes the existing value of the key, and should return the new value. If it returns None, no update is done and the operation is considered aborted. If not provided, the identity function is used.

Returns *:

The value returned by evaluator. If there’s an atomicity violation, None is returned.

Returns boolean:  

True if there’s no atomicity error. That is, this value is True if write succeeded or the user aborted, and False if there’s a atomicity violation (that prevented the write).

classmethod atomic_rw_hash(key, hash_key, evaluator=<function Redis.<lambda>>)

Atomically read-write a Redis hash key.

Parameters:

• key (string) – The key of the hash to read/write.
• hash_key (string) – The hash key within the hash to read/write.
• evaluator (function) – The evaluator function. It takes the existing value of the key, and should return the new value. If it returns None, no update is done and the operation is considered aborted. If not provided, the identity function is used.

Returns *:

The value returned by evaluator. If there’s an atomicity violation, None is returned.

Returns boolean:  

True if there’s no atomicity error. That is, this value is True if write succeeded or the user aborted, and False if there’s a atomicity violation (that prevented the write).

classmethod delete(key)

Delete the key from Redis.

Parameters:key (string) – The key to delete. Returns int:The number of items deleted. If 1, the key is found and deleted. If 0, the key is not found and nothing is done.

classmethod exists(key)

Whether the key exists in Redis.

Parameters:key (string) – The Redis key. Returns boolean:  True if the key exists, and False otherwise.

classmethod expire(key, second)

Set the key to expire in specified second.

Parameters:

• key (string) – The key to set expire.
• second (int) – The number of seconds for the key to live.

Returns boolean:  

True if the operation is successful.

classmethod get(key)

Get the value stored at the key.

Parameters:key (string) – The Redis key. Returns string:The value of the key. If the key does not exist, None will be returned.

classmethod get_json(key)

Get the value stored at the key as JSON.

Parameters:key (string) – The Redis key. Returns object:The value of the key as an unserialized JSON object. If the key does not exist, None will be returned.

classmethod hdelete(key, *hash_keys)

Delete keys from a hash table.

The hash_keys argument is a variable-length array and can be specified as follows:

redis.hdelete('danqing', 'key1', 'key2')
redis.hdelete('danqing', 'key2')
redis.hdelete('danqing', 'key1', 'key2', 'key3')

Parameters:

• key (string) – The key of the hash table.
• hash_keys (string...) – A list of hash keys to delete from the hash table.

Returns int:

The number of keys actually deleted. If 3 hash keys are specified but only 1 is found (and deleted), 1 is returned.

Raises:

redis.ResponseError – If key holds something other than a hash table.

classmethod hget(key, hash_key)

Get the value for a hash key in the hash table at the specified key.

Parameters:

• key (string) – The key to fetch.
• hash_key (string) – The hash key to fetch value for in the hash table.

Returns string:

The value corresponding to the hash key in the hash table. If either key or hash_key is not found, None is returned.

Raises:

redis.ResponseError – If key holds something other than a hash table.

classmethod hgetall(key)

Get the hash table at the specified key.

Parameters:key (string) – The key to fetch. Returns dict:The hash table at the specified key. If no hash table found (i.e. key not found), an empty dictionary is returned. Raises:redis.ResponseError – If key holds something other than a hash table.

classmethod hset(key, hash_key, hash_value)

Set the value for a hash key in the hash table at the specified key.

Parameters:

• key (string) – The key of the hash table. If the hash table does not exist yet, it will be created.
• hash_key (string) – The hash key to set value for in the hash table.
• hash_value (string) – The value to set to the key. If this is not a string, it will be casted to a string.

Returns int:

The number of new fields. If 1, a new field is added (hash_key is new). If 0, hash_key already exists and its value is updated.

Raises:

redis.ResponseError – If key holds something other than a hash table.

classmethod lpeek(key, count)

Peek the first count elements in the list without popping.

Parameters:

• key (string) – The key of the array.
• count (int) – The number of elements to peek.

Returns list:

The list of peeked elements.

classmethod lpop(key, count)

Pop the first count elements in the list.

Parameters:

• key (string) – The key of the array.
• count (int) – The number of elements to pop. If there are fewer than count elements, everything will be popped.

Returns list:

The list of popped elements.

classmethod rpush(key, *values)

Add values to a Redis list from the end.

The values argument is a variable-length array and can be specified as follows:

redis.rpush('danqing', 'val1', 'val2')
redis.rpush('danqing', 'val1')
redis.rpush('danqing', 'val1', 'val2', 'val3')

Parameters:

• key (string) – The key of the list. If the list does not exist yet, it will be created.
• values (string...) – A list of values to insert. If any is not a string, it will be casted to a string.

Returns int:

The total number of elements in the list after the push.

classmethod set(key, value)

Set the key to the specified value.

Parameters:

• key (string) – The Redis key.
• value (string) – The value to set. If this is not a string, it will be casted to a string.

Returns boolean:  

True if the operation is successful.

classmethod setex(key, value, second)

Set the key to the specified value, with an expiration time.

Parameters:

• key (string) – The Redis key.
• value (string) – The value to set.
• second (int) – The TTL in second.

Returns boolean:  

True if the operation is successful.

dq.redis.init_redis(key)

Initialize a Redis connection.

Parameters:key (string) – The config key. The entry should at least contain the host, port and db number of the instance. Returns redis:The redis instance if the config exists and is valid, and None otherwise.

dq.redis.strval(value)

JSON serialize value as appropriate.

This function should only be used internally.

Parameters:value (dict|list|string|number) – An input value. Returns string:The output value, suitable for saving by Redis. If value is a dict or list, it will be JSON-serialized. Otherwise it will be left as-is. Note that while Redis only takes string values, numbers have their string values be themselves in strings, and the conversion will be done by Redis automatically.

dq.redis.strvals(*values)

JSON serialize values as appropriate.

This function should only be used internally.

Parameters:values (..dict|list|string|number) – Input values. Returns list<string>:  The output values. See docs for strval for more explanations.

Retry

Retry helpers. A function can raise RecoverableError as needed so that it can be retried. The SQL variant can also be used to retry SQL errors.

dq.retry.recoverable_error(exception)

Use as @retry(retry_on_exception=recoverable_error).

dq.retry.sql_or_recoverable_error(exception)

Retries recoverable error and SQL errors.

This function rolls back the current session if the exception is a SQL one. Use as @retry(retry_on_exception=sql_or_recoverable_error).

String

String utilities that you may find useful.

dq.string.lower_no_punc(text)

Convert string to lowercase, stripping spaces and punctuations.

Parameters:text (string) – The text to clean up. Returns string:The cleaned up text.

dq.string.random_string(length)

Get a random string of lowercase letters + numbers of the given length.

Parameters:length (int) – The desired length of the string. Returns string:A random string of the given length.

dq.string.safe_relative_path(path)

Check whether a relative path is “safe”.

A path is safe if it is a relative path and doesn’t contain . or ..’s. This prevents us from directory traversal attacks.

Note that for simplicity, certain safe paths aren’t allowed either, notably … (e.g. …/path). Normal user should not really use such paths.

Parameters:path (string) – The path to check. Returns boolean:  True if the path is safe, and False otherwise.

dq.string.valid_filename(text)

Turn a string into a valid filename.

Parameters:text (string) – The string to convert. Returns string:A valid filename derived from the string.

Util

General utilities that you may find useful.

dq.util.safe_cast(value, to_type, default=None)

Cast a value to another type safely.

Parameters:

• value – The original value.
• to_type (type) – The destination type.
• default – The default value.