pycrate v0.7 releases: Python library to ease the development of encoders and decoders

by do son · Published February 25, 2019 · Updated November 21, 2023

Pycrate is a French word for qualifying bad wine. The present software library has nothing to do with bad wine, it is simply a Python library for manipulating various digital formats in an easy way. It is the glorious successor of libmich, which was started 8 years ago and served well.

Components

Pycrate is actually more a software suite than a single library. It is composed of several subdirectories, each providing specific services.

pycrate_core

The core of the library.

utils provides basics functions to manipulate integers, bytes and bits
charpy provides the Charpy class to handle easily the consumption of a bit-stream
elt and base are providing several classes to help when building complex data structures
repr provides simple functions to help with the representation of instances from the elt and base modules

Some of the most useful features are provided by the pack_val() functions from the utils module and the Charpy class from the charpy module. They help to deal easily with packing and unpacking bytes and integers (signed / unsigned, little / big endian) in an aligned and unaligned way. All lengths of fields are provided in bits, hence facilitating the handling of unaligned structures.

pycrate_ether

The modules provided here implement Ethernet and IP-oriented protocols and formats.

Ethernet with structures for Ethernet and VLAN headers
ARP simply providing the structure for ARP
IP with structures for IPv4, IPv6, ICMP, UDP and TCP
PCAP with structures for the PCAP global header and the record header

pycrate_media

The modules here implement various multimedia formats.

JPEG with detailed structures used in the JPEG format
GIF with detailed structures used in the GIF format
TIFF with detailed structures used in the TIFF format
BMP with structures used in the BMP format
PNG with the basic structure used in the PNG format
MPEG4 with the basic structure used in the MPEG4 file format
MP3 with detailed structures used in the MP3 format, including ID3v1 and ID3v2 tags

Most of the classes here implement a complete recipe to parse all of those formats in a single shot, by using their from_char()method.

pycrate_asn1c

All the modules here serve the sole purpose of compiling ASN.1 specifications. The most important ones are:

asnobj which is the almighty class when parsing any ASN.1 definition
generator which provides two distinct generators to produce source files from the ASN.1 objects processed in Python: PycrateGenerator which produces source file to be used with the pycrate ASN.1 runtime (in pycrate_asn1rt), and JSONDepGraphGenerator which produces json file listing object dependencies (which then can be browsed dynamically thanks to D3).
proc which is the top-level module for the compiler, it contains, for example, the compile_text() function which compiles a series of ASN.1 modules into Python objects

This compiler supports most of the ASN.1 language features, including parameterization and class objects and sets (especially useful when working with table constraints). It has, however, few restrictions, the biggest being the need for the left part of the ASN.1 assignment::= being on a single line. Also, old-school ASN.1 macros are not supported; hence, the compiler cannot parse SNMP MIBs.

pycrate_asn1dir

This subdirectory contains several ASN.1 specifications that are supported and precompiled for pycrate. Very few specifications have been changed in order to work with pycrate :

Q.775, in which the terrible AllPackagesAS is commented out
Q.773 and Q.775, in which the TCInvokeIdSet constraint is modified to be easier used as a set of values That’s all !

pycrate_asn1rt

This subdirectory contains the ASN.1 runtime, that is loaded and used by the ASN.1 specifications compiled with the compiler in pycrate_asn1c. It supports the PER encoding rules (aligned and not, canonical also), and the BER, CER and DER encoding rules.

pycrate_csn1

This subdirectory contains a CSN.1 to Python translator in the file trans.py, and a CSN.1 runtime in the file csnobj.py, in order to encode and decode CSN.1 structures translated to Python objects.

pycrate_csn1dir

This subdirectory contains some CSN.1 structures extracted from 3GPP specifications (in the .csn files), and translated into Python objects.

pycrate_mobile

This subdirectory implements most of the 3GPP NAS protocol formats:

GSMTAP: gsmtap header format
MCC_MNC: dictionnaries for MCC and MNC look-ups
NAS: provides two functions to parse any uplink and downlink mobile NAS messages
NASLTE: provides two functions to parse LTE uplink and downlink NAS messages
PPP: structures for NCP and LCP protocols used for PPP connection establishment
SCCP: structures for SCCP user-data and management messages
SIGTRAN: structures for SIGTRAN (mostly M2PA and M3UA) messages
TS23038: structures and routines for SMS encoding
TS23040_SMS: structures for the SMS transport protocol
TS23041_CBS: structures for the Cell Broadcast Service protocol
TS24007: basic structures from the TS 24.007 specification, reused in most of the NAS protocols
TS24008_CC : structures for call control messages from TS 24.008
TS24008_GMM: structures for GPRS mobility management messages from TS 24.008
TS24008_IE: structures for many information elements from TS 24.008
TS24008_MM: structures for mobility management messages from TS 24.008
TS24008_SM: structures for GPRS session management messages from TS 24.008
TS24011_PPSMS: structures for the SMS point-to-point protocol
TS24080_SS: structures for the Supplementary Services protocol, wrapping some MAP ASN.1 objects
TS24301_EMM: structures for the EPS mobility management messages from TS 24.301
TS24301_ESM: structures for the EPS mobility management messages from TS 24.301
TS24301_IE: structures for many information elements from TS 24.301
TS44018_GTTP: structure for the single GSM GTTP message from TS 44.018
TS44018_IE: structures for many information elements from TS 44.018
TS44018_RR: structures for the GSM and GPRS radio resources messages from TS 44.018

pycrate_corenet

This subdirectory implements a signalling server that supports an Iuh interface (including HNBAP and RUA/RANAP) for interfacing with 3G femtocells, and just enough of IuCS and IuPS stacks to support the attachment and mobility of mobiles connecting through the femtocells.

Changelog v0.7