dqpy¶
Danqing’s shared Python library, written with love for the better Danqing.
Cache¶
The cache module provides a simple JSON and Redis based caching decorator.
-
dq.cache.
cache
(ttl=600, key_func=None)[source]¶ Cache decorator.
This can be applied to any function that returns a raw or JSON-serializable response. To allow caching, the
cache
key must be set in the config, namely the redis connection for cache.If the function has a keyword argument named
fresh
, then the decorator gets a fresh copy when it’s set to a truthy value.If the function has a keyword argument named
raw
, then the decorator returns the raw (bytes) Redis response as-is, without JSON-deserializing.Parameters: - ttl (number) – The TTL in second. Default is 10 minutes.
- key_func (func) – The key function. This function should take the same arguments as the wrapped function, and return the corresponding cache key as a string.
Config¶
The config module helps getting configurations from TOML files stored in the config/
folder. The default file to use is config/local.toml
. To use another file, specify the DQENV
environment variable. For example, if DQENV=production
, then the config/production.toml
file will be used. Nested keys are comma-separated.
-
class
dq.config.
Config
[source]¶ Bases:
object
Configuration helper.
-
classmethod
get
(key)[source]¶ Get the config value specified by the provided key.
TOML config files should be stored in the config folder, and the environment (i.e. which file to choose) should be specified by the
DQENV
environment variable.Parameters: key (string) – The key to retrieve the value for. Nested key should have its components separated by dot, such as owner.email
.Returns: The value of the key if it exists, and None
otherwise.
-
classmethod
Database¶
SQL database client. This module provides connectors to the SQL database, with its configs specified (by you) in the config files. Check out the config/local.toml
file for an example configuration.
-
dq.database.
query_with_limit_offset
(query, limit, offset)[source]¶ Add limit and offset constraints to a SQLAlchemy query.
Parameters: - query (Query) – A SQLAlchemy query.
- limit (int) – The limit to add. If None or 0, it will be skipped.
- offset (int) – The offset to add. If None or 0, it will be skipped.
Returns Query: The updated query object.
-
dq.database.
replace_insert
(insert, compiler, **kw)[source]¶ Allow replace into for insert command.
This only works for MySQL. It’s not standard SQL. To enable this function, the sql.flavor config must be set to mysql.
-
dq.database.
safe_save_to_database
(model, session=None)[source]¶ Safely write things to the database.
This function simply calls
unsafe_save_to_database
under the hood, but returns whether the operation is successful instead of throwing an error upon failure.Parameters: - model (object/list) – The model object(s) to save. If a list is provided, the elements will be added one by one. Otherwise the object is added as-is.
- session (Session) – The optional database session to use.
Returns boolean: True if the operation is successful.
-
dq.database.
save_to_database
(model, session=None)[source]¶ Write things to the database.
This function simply calls
unsafe_save_to_database
under the hood, but wraps the error into aModelError
upon failure.Parameters: - model (object/list) – The model object(s) to save. If a list is provided, the elements will be added one by one. Otherwise the object is added as-is.
- session (Session) – The optional database session to use.
Raises: ModelError – if the save fails.
-
dq.database.
session_maker
()[source]¶ Generate a scoped session. In general, use the Session global variable.
Returns Session: A new SQLAlchemy session.
-
dq.database.
unsafe_save_to_database
(model, session=None)[source]¶ Convenient method for writing things to the database.
If the object is a list, its elements will be written to the database one by one. If the object is already in the database (i.e. when we are updating existing records), the caller of this function is responsible of ensuring that the same session used when retrieving the records is passed in. If a different session is provided, this operation will fail. (This is usually not an issue for view functions, which have session consistency on a thread level.)
Parameters: - model (object/list) – The model object(s) to save. If a list is provided, the elements will be added one by one. Otherwise the object is added as-is.
- session (Session) – The optional database session to use.
DBAdmin¶
This module is designed to be used as a shell script, for creating and dropping the database.
dbadmin create # Create a database with URL at sql.url
dbadmin drop # Drop the above database
dbadmin create another_sql.url # Create a database with URL at anothersql.url
Email¶
Module for sending emails.
-
dq.email.
send_email
(to, subject, html, sender_name, sender_email)[source]¶ Send an email.
Parameters: - to (string) – The recepient of the email.
- subject (string) – The subject of the email.
- html (string) – The HTML content of the email.
- sender_name (string) – The name of the sender.
- sender_email (string) – The sender address of the email.
Returns boolean: True if the email is sent successfully. False otherwise.
Entity¶
The entity module provide basic functionalities for Schematics entity classes.
-
class
dq.entity.
Entity
(raw_data=None, trusted_data=None, deserialize_mapping=None, init=True, partial=True, strict=True, validate=False, app_data=None, lazy=False, **kwargs)[source]¶ Bases:
schematics.deprecated.Model
Base entity class that provides shared utilities.
Currently it only allows printing a schematics entity. More utilities may be added in the future. In any case, all entities should inherit from this class.
Errors¶
The errors module contains a bunch of error classes so we don’t need to redefine them over and over. They also translate to HTTP errors nicely.
-
exception
dq.errors.
DQError
(message='', status_code=None, error_code=None)[source]¶ Bases:
Exception
Base exception.
Clients that wish to catch all exceptions should catch this one. The
status_code
will be set as the HTTP status code if a view handler triggers an error. If there is need to set the exact type of error, useerror_code
.4xx errors are due to the user, and as such the message will be passed directly back to user. 5xx errors will have the message replaced by the generic ‘Internal Error’ before sending back to user.
-
exception
dq.errors.
ForbiddenError
(message='Forbidden')[source]¶ Bases:
dq.errors.DQError
Exception raised when the user does not have the required permission.
-
exception
dq.errors.
IntegrationError
(message)[source]¶ Bases:
dq.errors.DQError
Exception raised when there is a service integration error.
-
exception
dq.errors.
InternalError
(message)[source]¶ Bases:
dq.errors.DQError
Exception raised when there is an internal server error.
-
exception
dq.errors.
InvalidInputError
(message='Invalid input')[source]¶ Bases:
dq.errors.DQError
Exception raised when the user input is not valid.
-
exception
dq.errors.
ModelError
(message)[source]¶ Bases:
dq.errors.DQError
Exception raised when something is wrong with the models.
-
exception
dq.errors.
NotFoundError
(message='Not found')[source]¶ Bases:
dq.errors.DQError
Exception raised when something is not found.
-
exception
dq.errors.
RecoverableError
(message)[source]¶ Bases:
dq.errors.DQError
Exception raised when a recoverable error occurs.
This may be due to network error, timeout, etc. and is meant to be retried. If retrying gives no luck, this is considered a 500 error.
Bases:
dq.errors.DQError
Exception raised when the user is not authenticated for an action.
FSUtil¶
The fsutil module contains file and file system helpers.
-
dq.fsutil.
fileinfo
(path)[source]¶ Information of a file.
This function throws an error if the path is not a valid file. To suppress errors, use
safe_fileinfo
instead.Parameters: path (string) – The path to the file. Returns string mime: The MIME type of the file, such as text/plain. Returns int size: The size (in bytes) of the file. Returns string encoding: The encoding of the file, like gzip. This is suitable for use as the Content-Encoding header.
-
dq.fsutil.
mkdirp
(path)[source]¶ Safely mkdir, creating all parent folders if they don’t yet exist.
This doesn’t raise an error if the folder already exists. However, it does raise
FileExistsError
if the path points to an existing file.Parameters: path (string) – The path to the folder to be created.
-
dq.fsutil.
rmrf
(path)[source]¶ Remove a path like rm -rf.
Parameters: path (string) – The path to remove.
-
dq.fsutil.
safe_fileinfo
(path)[source]¶ Retrive information of a file without throwing an error.
If the path is invalid, None, 0, None will be returned.
Parameters: path (string) – The path to the file. Returns string mime: The MIME type of the file, such as text/plain. Returns int size: The size (in bytes) of the file. Returns string encoding: The encoding of the file, like gzip. This is suitable for use as the Content-Encoding header.
-
dq.fsutil.
traverse
(path, callback)[source]¶ Traverse a directory recursively, performing a task.
Parameters: - path (string) – The path of the directory.
- callback (func) – A callback applied to each child file and directory. It takes 2 arguments, the relative path from the root, and whether the item is a directory.
FuncUtil¶
The funcutil module contains function helpers.
-
dq.funcutil.
safe_invoke
(noreturn=False)[source]¶ Invoke a function safely.
Parameters: noreturn (boolean) – Whether the function being called has no return value itself. Default to False. If True and the function returns successfully, True will be returned. None will be returned upon failure no matter whether the function has return value. Returns func: The decorator function that provides safe invocation.
Logging¶
The logging module helps format logging messages. It prints the message and pretty-prints the attached data dictionary. Use it as follows:
import logging
from dq.logging import error
logger = logging.getLogger(__name__)
error(logger, 'An error!', {'key': 'value'})
-
dq.logging.
debug
(logger, message, payload=None)[source]¶ Log a debug message. This does nothing in production enviornment.
Parameters: - logger (logger) – The logger to use.
- message (string) – The debug message.
- payload (dict) – The payload to attach to the log.
-
dq.logging.
error
(logger, message, payload=None)[source]¶ Log an error message.
Parameters: - logger (logger) – The logger to use.
- message (string) – The error message.
- payload (dict) – The payload to attach to the log.
ORM¶
The SQL ORM base classes. Your model classes should most likely inherit from IDBase
or UUIDBase
, and some other mixins.
-
class
dq.orm.
Attr
(name, allow_from, allow_to, serializer, deserializer)¶ Bases:
tuple
-
allow_from
¶ Alias for field number 1
-
allow_to
¶ Alias for field number 2
-
deserializer
¶ Alias for field number 4
-
name
¶ Alias for field number 0
-
serializer
¶ Alias for field number 3
-
-
class
dq.orm.
DictMixin
[source]¶ Bases:
object
Dictionary mixin that provides converters to/from Python dicts.
See comments of
Attr
above for defining fields to convert with this mixin.This class is inherited by
IDBase
andUUIDBase
classes, so your class will not need to inherit directly from it.-
classmethod
from_dict
(data)[source]¶ Convert a dictionary to an object.
Parameters: data (dict) – The data dictionary. Returns Base: The object converted from the dictionary.
-
inflate_to_dict
(session=None)[source]¶ Convert an object to dictionary, with relationships inflated.
This method is the default implementation, which simply calls the
to_dict
method. Subclasses willing to inflate certain fields should override this method.Parameters: session (Session) – The optional SQL session. Returns dict: The dictionary converted from the object.
-
classmethod
-
class
dq.orm.
QueryMixin
[source]¶ Bases:
object
Query helper functions useful to all models.
This class is inherited by
IDBase
andUUIDBase
classes, so your class will not need to inherit directly from it.-
classmethod
get_by
(column, value, for_update=False, contains_deleted=False, contains_empty=False, session=None)[source]¶ Get the object satisfying the query condition.
Parameters: - column (string) – The name of the column to query by.
- value (string) – The value of the column to query for.
- for_update (boolean) – Whether the query is for updating the row.
- contains_deleted (boolean) – Whether to contain deleted records.
Default is
False
and only active records are returned. - contains_empty (boolean) – Whether to contain empty records.
Default is
False
and if value isNone
,None
will be returned. - session (Session) – Optional SQL session.
Returns QueryMixin: The matching object. This method is designed for unique queries and in case of multiple matches, only the first one is returned.
-
classmethod
get_multi
(column, value, sort_column='updated_at', desc=True, offset=0, limit=10, session=None)[source]¶ Get multiple objects satisfying the query condition.
Parameters: - column (string) – The name of the column to query by.
- value (string) – The value of the column to query for.
- sort_column (string) – The column to sort results against. Default is updated_at. Please note that created_at usually doesn’t have index, so if it should be used as the sort column, an index needs to be added.
- desc (bool) – Whether to sort DESC. Default is
True
. - offset (int) – The query offset. Default is 0.
- limit (int) – The max number of results to return. Default is 10. If specified, the function will try to fetch one more result to indicate whether there are more results. If not specified or 0, all results will be returned.
- session (Session) – The optional SQL session to use.
-
classmethod
-
class
dq.orm.
TimeMixin
[source]¶ Bases:
object
Time fields common to all models.
This class is inherited by
IDBase
andUUIDBase
classes, so your class will not need to inherit directly from it.-
created_at
= Column(None, ArrowType(), table=None, nullable=False, default=ColumnDefault(<function utcnow>))¶
-
deleted_at
= Column(None, ArrowType(), table=None)¶
-
updated_at
= Column(None, ArrowType(), table=None, nullable=False, onupdate=ColumnDefault(<function utcnow>), default=ColumnDefault(<function utcnow>))¶
-
-
class
dq.orm.
UserRelationMixin
[source]¶ Bases:
dq.orm.QueryMixin
Mixin that provides a common
get_by_user
function.This mixin should be used by classes that a user can have a 1-many relationship to, such as orders, shipments, and periods.
-
classmethod
get_by_user
(user_id, desc=True, offset=0, limit=10, session=None)[source]¶ Get the related object of a user.
Parameters: - user_id (string/int) – The ID (if int) or UUID (if string) of the user.
- desc (bool) – Whether to sort DESC. Default is
True
. - offset (integer) – The offset of the objects. If limit is 10 and offset is 5, the function will attempt to get the 6th to 15th most recent objects.
- limit (integer) – The max number of objects to get.
- session (Session) – The optional SQL session.
Returns list<UserRelationMixin>: The list of objects matching the requirement.
-
sort_key
= None¶
-
classmethod
-
dq.orm.
arrow_in
(value)[source]¶ Convert an arrow-understandable time to an Arrow object.
Parameters: value – The arrow-understandable time or None
.Returns Arrow: The Arrow object, or None
if the input isNone
.
-
dq.orm.
arrow_out
(value)[source]¶ Convert an Arrow object to timestamp.
Parameters: value (Arrow) – The Arrow time or None
.Returns int: The timestamp, or None
if the input isNone
.
-
dq.orm.
boolean_mask
(value)[source]¶ Mask a value to boolean. This can be used for sensitive fields.
Parameters: value – Any value. Returns boolean: The input value casted to boolean.
Redis¶
The Redis client. It reads the config files for Redis configs, which you must specify to use this client. Refer to the config/local.toml
file in this repo for an example.
-
class
dq.redis.
Redis
[source]¶ Bases:
object
-
classmethod
atomic_rw
(key, evaluator=<function Redis.<lambda>>)[source]¶ Atomically read-write a Redis key.
Parameters: - key (string) – The key to read/write.
- evaluator (function) – The evaluator function. It takes the existing value of the key, and should return the new value. If it returns None, no update is done and the operation is considered aborted. If not provided, the identity function is used.
Returns *: The value returned by evaluator. If there’s an atomicity violation, None is returned.
Returns boolean: True if there’s no atomicity error. That is, this value is True if write succeeded or the user aborted, and False if there’s a atomicity violation (that prevented the write).
-
classmethod
atomic_rw_hash
(key, hash_key, evaluator=<function Redis.<lambda>>)[source]¶ Atomically read-write a Redis hash key.
Parameters: - key (string) – The key of the hash to read/write.
- hash_key (string) – The hash key within the hash to read/write.
- evaluator (function) – The evaluator function. It takes the existing value of the key, and should return the new value. If it returns None, no update is done and the operation is considered aborted. If not provided, the identity function is used.
Returns *: The value returned by evaluator. If there’s an atomicity violation, None is returned.
Returns boolean: True if there’s no atomicity error. That is, this value is True if write succeeded or the user aborted, and False if there’s a atomicity violation (that prevented the write).
-
classmethod
delete
(key)[source]¶ Delete the key from Redis.
Parameters: key (string) – The key to delete. Returns int: The number of items deleted. If 1, the key is found and deleted. If 0, the key is not found and nothing is done.
-
classmethod
exists
(key)[source]¶ Whether the key exists in Redis.
Parameters: key (string) – The Redis key. Returns boolean: True
if the key exists, andFalse
otherwise.
-
classmethod
expire
(key, second)[source]¶ Set the key to expire in specified second.
Parameters: - key (string) – The key to set expire.
- second (int) – The number of seconds for the key to live.
Returns boolean: True if the operation is successful.
-
classmethod
get
(key)[source]¶ Get the value stored at the key.
Parameters: key (string) – The Redis key. Returns string: The value of the key. If the key does not exist, None
will be returned.
-
classmethod
get_json
(key)[source]¶ Get the value stored at the key as JSON.
Parameters: key (string) – The Redis key. Returns object: The value of the key as an unserialized JSON object. If the key does not exist, None
will be returned.
-
classmethod
hdelete
(key, *hash_keys)[source]¶ Delete keys from a hash table.
The
hash_keys
argument is a variable-length array and can be specified as follows:redis.hdelete('danqing', 'key1', 'key2') redis.hdelete('danqing', 'key2') redis.hdelete('danqing', 'key1', 'key2', 'key3')
Parameters: - key (string) – The key of the hash table.
- hash_keys (string...) – A list of hash keys to delete from the hash table.
Returns int: The number of keys actually deleted. If 3 hash keys are specified but only 1 is found (and deleted), 1 is returned.
Raises: redis.ResponseError – If
key
holds something other than a hash table.
-
classmethod
hget
(key, hash_key)[source]¶ Get the value for a hash key in the hash table at the specified key.
Parameters: - key (string) – The key to fetch.
- hash_key (string) – The hash key to fetch value for in the hash table.
Returns string: The value corresponding to the hash key in the hash table. If either
key
orhash_key
is not found,None
is returned.Raises: redis.ResponseError – If
key
holds something other than a hash table.
-
classmethod
hgetall
(key)[source]¶ Get the hash table at the specified key.
Parameters: key (string) – The key to fetch. Returns dict: The hash table at the specified key. If no hash table found (i.e. key not found), an empty dictionary is returned. Raises: redis.ResponseError – If key
holds something other than a hash table.
-
classmethod
hset
(key, hash_key, hash_value)[source]¶ Set the value for a hash key in the hash table at the specified key.
Parameters: - key (string) – The key of the hash table. If the hash table does not exist yet, it will be created.
- hash_key (string) – The hash key to set value for in the hash table.
- hash_value (string) – The value to set to the key. If this is not a string, it will be casted to a string.
Returns int: The number of new fields. If 1, a new field is added (
hash_key
is new). If 0,hash_key
already exists and its value is updated.Raises: redis.ResponseError – If
key
holds something other than a hash table.
-
classmethod
lpeek
(key, count)[source]¶ Peek the first count elements in the list without popping.
Parameters: - key (string) – The key of the array.
- count (int) – The number of elements to peek.
Returns list: The list of peeked elements.
-
classmethod
lpop
(key, count)[source]¶ Pop the first count elements in the list.
Parameters: - key (string) – The key of the array.
- count (int) – The number of elements to pop. If there are fewer
than
count
elements, everything will be popped.
Returns list: The list of popped elements.
-
classmethod
rpush
(key, *values)[source]¶ Add values to a Redis list from the end.
The
values
argument is a variable-length array and can be specified as follows:redis.rpush('danqing', 'val1', 'val2') redis.rpush('danqing', 'val1') redis.rpush('danqing', 'val1', 'val2', 'val3')
Parameters: - key (string) – The key of the list. If the list does not exist yet, it will be created.
- values (string...) – A list of values to insert. If any is not a string, it will be casted to a string.
Returns int: The total number of elements in the list after the push.
-
classmethod
-
dq.redis.
init_redis
(key)[source]¶ Initialize a Redis connection.
Parameters: key (string) – The config key. The entry should at least contain the host, port and db number of the instance. Returns redis: The redis instance if the config exists and is valid, and None otherwise.
-
dq.redis.
strval
(value)[source]¶ JSON serialize value as appropriate.
This function should only be used internally.
Parameters: value (dict|list|string|number) – An input value. Returns string: The output value, suitable for saving by Redis. If value
is adict
orlist
, it will be JSON-serialized. Otherwise it will be left as-is. Note that while Redis only takes string values, numbers have their string values be themselves in strings, and the conversion will be done by Redis automatically.
Retry¶
Retry helpers. A function can raise RecoverableError
as needed so that it can be retried. The SQL variant can also be used to retry SQL errors.
String¶
String utilities that you may find useful.
-
dq.string.
lower_no_punc
(text)[source]¶ Convert string to lowercase, stripping spaces and punctuations.
Parameters: text (string) – The text to clean up. Returns string: The cleaned up text.
-
dq.string.
random_string
(length)[source]¶ Get a random string of lowercase letters + numbers of the given length.
Parameters: length (int) – The desired length of the string. Returns string: A random string of the given length.
-
dq.string.
safe_relative_path
(path)[source]¶ Check whether a relative path is “safe”.
A path is safe if it is a relative path and doesn’t contain . or ..’s. This prevents us from directory traversal attacks.
Note that for simplicity, certain safe paths aren’t allowed either, notably … (e.g. …/path). Normal user should not really use such paths.
Parameters: path (string) – The path to check. Returns boolean: True if the path is safe, and False otherwise.