YSK

Articles to read from (and take notes):

• Article: https://avi.im/blag/2021/fast-sqlite-inserts/

  PyPy is really fast implementation of Python. (Recommended to be used under any case)

Some Python Libraries

• pycurl: PycURL is python interface to libcurl. PycURL can be used to fetch objects identified by a URL from a Python program, similar to urllib Python module. Pycurl is mature, very fast and has lot of features.

  • libcurl is a free and easy-to-use client-side URL transfer library, supporting FTP, FTPS, HTTP, HTTPS, SCP, SFTP, TFTP, TELNET, DICT, LDAP, LDAPS, FILE, IMAP, SMTP, POP3 and RTSP. libcurl supports SSL certificates, HTTP POST, HTTP PUT, FTP uploading, HTTP form based upload, proxies, cookies, user+password authentication (Basic, Digest, NTLM, Negotiate, Kerberos4), file transfer resume, http proxy tunneling and more!

• safety - reports vulnerabilities for Python.

  • Installation: pip3 install safety
  • Check dependencies: safety check -r requirements.txt

• glom - working with data in python - Path based access for nested structures - declarative data transformation using lightweight - Pythonic specifications. (https://glom.readthedocs.io/en/latest/)

  • Readable error messages, built in debugging and deep assignment, streaming and data validation.

  from glom import glom
  
  target = {'a': {'b': {'c': 'd'}}}
  glom(target, 'a.b.c')  # returns 'd'
  

  glom is very powerful and is a must to use with APIs and JSON. Here's the library: https://glom.readthedocs.io/en/latest/tutorial.html

  • There are even options to specify fallback to destructing json to a target using Coalesce

  >>> from glom import Coalesce
  >>> target = {
  ...     'system': {
  ...         'planets': [
  ...             {'name': 'earth', 'moons': 1},
  ...             {'name': 'jupiter', 'moons': 69}
  ...         ]
  ...     }
  ... }
  >>> spec = {
  ...     'planets': (Coalesce('system.planets', 'system.dwarf_planets'), ['name']),
  ...     'moons': (Coalesce('system.planets', 'system.dwarf_planets'), ['moons'])
  ... }
  >>> pprint(glom(target, spec))
  {'moons': [1, 69], 'planets': ['earth', 'jupiter']}
  

  Logging in Python

  Logging in JSON

  • python-json-logger: The most lightweight python for json logging.
  • structlog: https://www.structlog.org/en/stable/
  • loguru: https://github.com/Delgan/loguru
  • From docs, the logging cookbook: https://docs.python.org/3/howto/logging-cookbook.html#implementing-structured-logging

  Compactness of Shell Scripts in Python - Plumbum

  (https://plumbum.readthedocs.io/en/latest/index.html)

  Apart from shell-like syntax and handy shortcuts, the library provides local and remote command execution (over SSH), local and remote file-system paths, easy working-directory and environment manipulation, quick access to ANSI colors, and a programmatic Command-Line Interface (CLI) application toolkit. Now let’s see some code!

  >>> ls = local["ls"]
  >>> ls
  LocalCommand(<LocalPath /bin/ls>)
  >>> ls()
  u'build.py\\ndist\\ndocs\\nLICENSE\\nplumbum\\nREADME.rst\\nsetup.py\\ntests\\ntodo.txt\\n'
  >>> notepad = local["c:\\\\windows\\\\notepad.exe"]
  >>> notepad()                                   # Notepad window pops up
  u''                                             # Notepad window is closed by user, command returns
  

  >>> chain = ls["-a"] | grep["-v", "\\\\.py"] | wc["-l"]
  >>> print chain
  /bin/ls -a | /bin/grep -v '\\.py' | /usr/bin/wc -l
  >>> chain()
  u'13\\n'
  

Connexion from Zalando

• https://github.com/zalando/connexion - Generate python flask with just the API specification.
• Hug: Does opposite of what Connexion does - Generate API spec from the API implementation: https://github.com/timothycrosley/hug
• https://engineering.zalando.com/posts/2016/12/crafting-effective-microservices-in-python.html

Python - local webserver to display current server:

python -m SimpleHTTPServer

Tox - Automation Project - Standardize testing in Python - Something that Brainzutils might use in future.