It is a new selection of tips and tricks about Python and programming from my Telegram-channel @pythonetc.
Previous publications.
Storing and sending object via network as bytes is a huge topic. Let’s discuss some tools that are usually used for that in Python and their advantages and disadvantages.
As an example I’ll try to serialize the Cities object which contains some City objects as well as their order. Here is four method you can use:
1. JSON. It’s human readable, easy to use, but consumes a lot of memory. The same is true for other formats like YAML or XML.
class City:
def to_dict(self):
return dict(
name=self._name,
country=self._country,
lon=self._lon,
lat=self._lat,
)
class Cities:
def __init__(self, cities):
self._cities = cities
def to_json(self):
return json.dumps([
c.to_dict() for c in self._cities
]).encode('utf8')2. Pickle. Pickle is native for Python, can be customized and consumes less memory than JSON. The downside is you have to use Python to unpickle the data.
class Cities:
def pickle(self):
return pickle.dumps(self)3. Protobuf (and other binary serializers such as msgpack). Consumes even less memory, can be used from any other programming languages, but require custom schema:
syntax = "proto2";
message City {
required string name = 1;
required string country = 2;
required float lon = 3;
required float lat = 4;
}
message Cities {
repeated City cities = 1;
}
class City:
def to_protobuf(self):
result = city_pb2.City()
result.name = self._name
result.country = self._country
result.lon = self._lon
result.lat = self._lat
return result
class Cities:
def to_protobuf(self):
result = city_pb2.Cities()
result.cities.extend([
c.to_protobuf() for c in self._cities
])
return result4. Manual. You can manually pack and unpack data with the struct module. It allow you to consume the absolute minimum amount of memory, but protobuf still can be a better choice since it supports versioning and explicit schemas.
class City:
def to_bytes(self):
name_encoded = self._name.encode('utf8')
name_length = len(name_encoded)
country_encoded = self._country.encode('utf8')
country_length = len(country_encoded)
return struct.pack(
'BsBsff',
name_length, name_encoded,
country_length, country_encoded,
self._lon, self._lat,
class Cities:
def to_bytes(self):
return b''.join(
c.to_bytes() for c in self._cities
)
If a function argument has the default value of
None and is annotated as T, mypy automatically treats it as Optional[T] (in other words, Union[T, None]).That doesn't work with other types, so you can't have something like
f(x: A = B()). It also doesn't work with a variable assignment: a: A = None will cause an error.def f(x: int = None):
reveal_type(x)
def g(y: int = 'x'):
reveal_type(y)
z: int = None
reveal_type(z)
$ mypy test.py
test.py:2: error: Revealed type is 'Union[builtins.int, None]'
test.py:4: error: Incompatible default for argument "y" (default has type "str", argument has type "int")
test.py:5: error: Revealed type is 'builtins.int'
test.py:7: error: Incompatible types in assignment (expression has type "None", variable has type "int")
test.py:8: error: Revealed type is 'builtins.int'In Python 3, once the
except block is exited, the variables that store caught exceptions are removed from locals() even if they previously existed:>>> e = 2
>>> try:
... 1/0
... except Exception as e:
... pass
...
>>> e
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
NameError: name 'e' is not definedIf you want to save a reference to the exception, you have to use another variable:
>>> error = None
>>> try:
... 1/0
... except Exception as e:
... error = e
...
>>> error
ZeroDivisionError('division by zero',)This is not true for Python 2.
You may have your own
pypi repository. It lets you release packages inside your project and install them with pip as though they are regular packages.It is remarkable that you don’t have to install any specific software, but can use a regular http-server instead. Here is how it works for me, for example.
Let’s have a trivial package named
pythonetc.setup.py:
from setuptools import setup, find_packages
setup(
name='pythonetc',
version='1.0',
packages=find_packages(),
)
pythonetc.py:
def ping():
return 'pong'Let’s release it to the ~/pypi directory:
$ python setup.py sdist bdist_wheel
…
$ mv dist ~/pypi/pythonetcNow server this on the
pypi.pushtaev.ru domain with nginx:$ cat /etc/nginx/sites-enabled/pypi
server {
listen 80;
server_name pypi.pushtaev.ru;
root /home/vadim/pypi;
index index.html index.htm index.nginx-debian.html;
location / {
autoindex on;
try_files $uri $uri/ =404;
}
}It now can be installed:
$ pip install -i http://pypi.pushtaev.ru --trusted-host pypi.pushtaev.ru pythonetc
…
Collecting pythonetc
Downloading http://pypi.pushtaev.ru/pythonetc/pythonetc-1.0-py3-none-any.whl
Installing collected packages: pythonetc
Successfully installed pythonetc-1.0
$ python
Python 3.7.0+ (heads/3.7:0964aac, Mar 29 2019, 00:40:55)
[GCC 4.9.2] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pythonetc
>>> pythonetc.ping()
'pong'It’s quite often when you have to declare a dictionary with all keys equal to the local variables with the same name. Something like this:
dict(
context=context,
mode=mode,
action_type=action_type,
)
ECMAScript even has the special form of object literal for such cases (it’s called Object Literal Property Value Shorthand):
> var a = 1;
< undefined
> var b = 2;
< undefined
> {a, b}
< {a: 1, b: 2}It is possible to create a similar helper in Python (alas, it looks not even closely as good as the ECMAScript notation):
def shorthand_dict(lcls, names):
return {k: lcls[k] for k in names}
context = dict(user_id=42, user_ip='1.2.3.4')
mode = 'force'
action_type = 7
shorthand_dict(locals(), [
'context',
'mode',
'action_type',
])You may wonder why we have to pass
locals() as a parameter in the previous example. Is it possible to get the locals of the caller in the callee? It is indeed, but you have to mess with the inpsect module:import inspect
def shorthand_dict(names):
lcls = inspect.currentframe().f_back.f_locals
return {k: lcls[k] for k in names}
context = dict(user_id=42, user_ip='1.2.3.4')
mode = 'force'
action_type = 7
shorthand_dict([
'context',
'mode',
'action_type',
])You can go even further and use something like this — https://github.com/alexmojaki/sorcery:
from sorcery import dict_of
dict_of(context, mode, action_type)
