# Python Deep Drive III 第 7 節 – Serialization and Deserialization (43~49)

**URL:** <https://vip.studycamp.tw/t/python-deep-drive-iii-%E7%AC%AC-7-%E7%AF%80-%E2%80%93-serialization-and-deserialization-4349/5298>\
**Category:** 3. Dictionaries\
**Created:** [2022年十二月16日 11:17 UTC](https://vip.studycamp.tw/t/python-deep-drive-iii-%E7%AC%AC-7-%E7%AF%80-%E2%80%93-serialization-and-deserialization-4349/5298 "2022-12-16T11:17:47Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![sky](https://vip.studycamp.tw/user_avatar/vip.studycamp.tw/sky/32/8098_2.png) [@sky](https://vip.studycamp.tw/u/sky)\
**Post date:** [2022年十二月16日 11:17 UTC](https://vip.studycamp.tw/t/python-deep-drive-iii-%E7%AC%AC-7-%E7%AF%80-%E2%80%93-serialization-and-deserialization-4349/5298/1 "2022-12-16T11:17:47Z")

</div>

## pickling

### pickling on UNIX

> **重要！unpickle 可執行程式，有資安風險。不要 unpickling 不是你自己 pickling 的資料。**

```python
import os
import pickle

class Exploit():
    def __reduce__ (self):
        # cat [-u] [-s] [-v [-t] [-e] ] file 本例用來檢視內容，並存入 exploit.txt
        # curl [options] [URL...] 本例取得 google 首頁資料，也是存入 exploit.txt
        return (os.system, ("cat /etc/passwd > exploit.txt && curl www.google.com >> exploit.txt",))

def serialize_exploit(fname):
    with open(fname, 'wb') as f: # wb: write binary
        pickle.dump(Exploit(), f)

serialize_exploit('loadme')

pickle.load(open('loadme', 'rb')) # rb: read binary

```

然後觀察該路徑（`/etc/passwd`）的相關檔案變化，以及檔案內容（`exploit.txt`，基本上就是上方程式中註解說明的：/etc/passwd 路徑內容，加上 google 首頁資料）。

> 上述程式僅支援 UNIX & iOS，不支援 MS-Windows。

### Pickling Dictionaries

```python
import pickle

ser = pickle.dumps('Python Pickled Peppers')
ser
# b'\x80\x04\x95\x1a\x00\x00\x00\x00\x00\x00\x00\x8c\x16Python Pickled Peppers\x94.'

deser = pickle.loads(ser)
deser
# 'Python Pickled Peppers'

# 精簡過，非原整原始碼。
# 先指定 d = 原始資料，再令 er = pickle.dumps(d)，藉此來展示 is(identiy) 和 ==(equality)
d is deser, d == deser
# (False, True)

# 接著展示不同資料型態的執行結果
# ser = pickle.dumps(3.14) # float
# ser = pickle.dumps([10, 20, ('a', 'b', 30)]) # list
# s = {'a', 'b', 'x', 10} # set
# ser = pickle.dumps({'b': 1, 'a': 2, 'c': {'x': 10, 'y': 20}}) # dictionary

```

如果要 pickle 的 dictionary，其中部分值是參考另一個 dictionary 呢？

```python
d1 = {'a': 10, 'b': 20}
d2 = {'x': 100, 'y': d1, 'z': d1}

ser = pickle.dumps(d2)
d3 = pickle.loads(ser)

d3['y'] == d2['y']
# True

d3['y'] is d2['y']
# False

```

接著看如果資料來源是同一個（上述 d1）時，pickling 不會再做一次，而是指向同一個物件（類似 deepcopy 的行為模式）。

不只是 dictionary，lists, sets 和 tuples 也一樣。

```python
d2['y'] is d2['z']
# True

d3['y'] == d3['z']
# True

```

上述行為，在我們使用 serialization / deserialization 時要特別小心。因為參考同一個 object，serialization / deserialization 後的值，會是原來的參考值。

> 對照一：沒有用 serialization 時，行為符合我們預期

```python
d1 = {'a': 1, 'b': 2}
d2 = {'x': 10, 'y': d1}
print(d1)
# {'a': 1, 'b': 2}
print(d2)
# {'x': 10, 'y': {'a': 1, 'b': 2}}

d1['c'] = 3
print(d1)
# {'a': 1, 'b': 2, 'c': 3}
print(d2)
# {'x': 10, 'y': {'a': 1, 'b': 2, 'c': 3}}

```

> 對照二：使用 serialization 時。下次程式執行時，d2 的值，對照到原先的 d1（沒有 ‘c’: 3），而不是新的。

```python
d1 = {'a': 1, 'b': 2}
d2 = {'x': 10, 'y': d1}
d1_ser = pickle.dumps(d1)
d2_ser = pickle.dumps(d2)

# simulate exiting the program, or maybe just restarting the notebook
del d1
del d2

# load the data back up
d1 = pickle.loads(d1_ser)
d2 = pickle.loads(d2_ser)

# and continue processing as before
print(d1)
# {'a': 1, 'b': 2}
print(d2)
# {'x': 10, 'y': {'a': 1, 'b': 2}}

d1['c'] = 3
print(d1)
# {'a': 1, 'b': 2, 'c': 3}
print(d2)
# {'x': 10, 'y': {'a': 1, 'b': 2}} # 這裡沒有 'c': 3

```

最後用老範例（class Person），再驗證一次上述行為。

> The pickle module is relatively intelligent and will not re-pickle an object it has already pickled - which means that **relative** references are preserved.

\_\_

## JSON Serialization

### 基本操作

```python
import json # 注意！本例下方的範例程式，已省略本 import

# serialization
d1 = {'a': 100, 'b': 200}
d1_json = json.dumps(d1)
d1_json, type(d1_json)
# ('{"a": 100, "b": 200}', str)

# deserialization
d2 = json.loads(d1_json)
d2, type(d2)
# ({'a': 100, 'b': 200}, dict)

# equality checking
d1 == d2
# True

```

> 提醒：d1 serialization 前後的不同。 {‘a’: 100, ‘b’: 200} vs ‘{“a”: 100, “b”: 200}’

我們可以在 print 中，設定 json.dumps 的參數，讓顯示美觀些。

```python
print(json.dumps(d1, indent=2))

```

輸出：

```python
{
  "a": 100,
  "b": 200
}

```

### 注意資料型態

**Python 的 key，只要是 hashable 即可。但 JSON 的 key，必須是 string。**

先來看看當 Python 的 key 是整數時，程式執行過程。

```python
d1 = {1: 100, 2: 200}
d1_json = json.dumps(d1)
d1_json
# '{"1": 100, "2": 200}' # from 1 to "1"

d2 = json.loads(d1_json)
print(d1)
# {1: 100, 2: 200}
print(d2)
# {'1': 100, '2': 200}

```

> 注意：上方的程式顯示，Python 將 key 整數，轉換為 string。1 變 “1”、2 變 “2”。
> 
> 所以 d == loads(dumps(d)) 不成立。

接著看一下，Python 中的一個長字串（內容是 JSON），在做過 serialization / deserialization 後的結果。

```python
d_json = '''
{
    "name": "John Cleese",
    "age": 82,
    "height": 1.96,
    "walksFunny": true,
    "sketches": [
        {
        "title": "Dead Parrot",
        "costars": ["Michael Palin"]
        },
        {
        "title": "Ministry of Silly Walks",
        "costars": ["Michael Palin", "Terry Jones"]
        }
    ],
    "boring": null    
}
'''

```

```python
# deserialize this JSON string
d = json.loads(d_json)
print(d)
# {'name': 'John Cleese', 'age': 82, 'height': 1.96,... 過長，恕刪

d
{'name': 'John Cleese',
 'age': 82,
 'height': 1.96,
 'walksFunny': True, # 原本是 true
 'sketches': [{'title': 'Dead Parrot', 'costars': ['Michael Palin']},
  {'title': 'Ministry of Silly Walks',
   'costars': ['Michael Palin', 'Terry Jones']}],
 'boring': None} # 原本是 null

```

> 注意：serialization / deserialization 後，JSON object不保證 key 的順序相同。

接著檢查 serialization / deserialization 後的資料型態。

```python
print(d['age'], type(d['age']))
print(d['height'], type(d['height']))
print(d['boring'], type(d['boring']))
print(d['sketches'], type(d['sketches']))
print(d['walksFunny'], type(d['walksFunny']))
print(d['sketches'][0], type(d['sketches'][0]))

```

輸出：

```python
82 <class 'int'>
1.96 <class 'float'>
None <class 'NoneType'>
[{'title': 'Dead Parrot', 'costars': ['Michael Palin']}, {'title': 'Ministry of Silly Walks', 'costars': ['Michael Palin', 'Terry Jones']}] <class 'list'>
True <class 'bool'>
{'title': 'Dead Parrot', 'costars': ['Michael Palin']} <class 'dict'>

```

As you can see the JSON  
`array` was serialized into a `list` ,  
`true` was serialized into a `bool` ,  
integer looking values into `int` ,  
float looking values into `float` and  
sub-objects into `dict` .

As you can see deserializing JSON objects into Python is very straightforward and intuitive.

### tuples serializing

```python
d = {'a': (1, 2, 3)}
json.dumps(d)
# '{"a": [1, 2, 3]}'

```

> Python tuples are serialized into JSON lists

```python
bad_json = '''
    {"a": (1, 2, 3)}
'''

json.loads(bad_json)
# JSONDecodeError # 這個錯誤提示不是太明顯

```

> Python was able to serialize a tuple by making it into a JSON array
> 
> How about other data types: Decimals, Fractions, Complex Numbers, Sets, etc

### Decimals

```python
from decimal import Decimal
json.dumps({'a': Decimal('0.5')})
# TypeError 

```

### Complex

```python
try:
    json.dumps({"a": 1+1j})
except TypeError as ex:
    print(ex)

# Object of type 'complex' is not JSON serializable

```

### Set

```python
try:
    json.dumps({"a": {1, 2, 3}})
except TypeError as ex:
    print(ex)

# Object of type 'set' is not JSON serializable

```

> Now we could get around that problem by looking at the string representation of those objects:

```python
str(Decimal(0.5))
# '0.5'

json.dumps({"a": str(Decimal(0.5))})
# '{"a": "0.5"}'

```

### Own objects

```python
class Person:
    def __init__ (self, name, age):
        self.name = name
        self.age = age
    
    def __repr__ (self):
        return f'Person(name={self.name}, age={self.age})'

p = Person('John', 82)
p
# Person(name=John, age=82)

json.dumps({"john": p})
# TypeError

```

> Solution: write a custom JSON serializer in our class itself, and use that when we serialize the object

```python
class Person:
    def __init__ (self, name, age):
        self.name = name
        self.age = age
    
    def __repr__ (self):
        return f'Person(name={self.name}, age={self.age})'
    
    def toJSON(self):
        return dict(name=self.name, age=self.age)
p = Person('John', 82)
p.toJSON()
# {'name': 'John', 'age': 82}

print(json.dumps({"john": p.toJSON()}, indent=2))

```

輸出：

```python
{
  "john": {
    "name": "John",
    "age": 82
  }
}

```

> we can make our life a little easier by using the `vars` function (or the ` __dict__ ` attribute) to return a dictionary of our object attributes

```python
vars(p) # vars function
# {'name': 'John', 'age': 82}

p. __dict__
# {'name': 'John', 'age': 82}

class Person:
    def __init__ (self, name, age):
        self.name = name
        self.age = age
    
    def __repr__ (self):
        return f'Person(name={self.name}, age={self.age})'
    
    def toJSON(self):
        # return dict(name=self.name, age=self.age)
        return vars(self)

json.dumps(dict(john=p.toJSON()))
# '{"john": {"name": "John", "age": 82}}'

```

> dealing with sets, where we do not control the class definition

```python
s = {1, 2, 3}

json.dumps(dict(a=list({1, 2, 3})))
# '{"a": [1, 2, 3]}'

```

問題：

1. we have to remember to call `.toJSON()` for our custom objects

2. what about built-in or standard types like sets, or dates? use built-in or write custom functions to convert and call them every time?

* * *

### Custom JSON Serialization

#### datetime

```python
from datetime import datetime
import json

current = datetime.utcnow()
# datetime.datetime(2022, 12, 20, 11, 37, 44, 408640)

json.dumps(current)
# TypeError

```

> Python raises a `TypeError` exception, stating that `datetime` objects are not JSON serializable.

```python
from datetime import datetime
import json

def format_iso(dt):
    return dt.strftime('%Y-%m-%dT%H:%M:%S')
format_iso(current)
# '2022-12-20T11:37:44'

current.isoformat()
# '2022-12-20T11:37:44.408640'

log_record = {'time': datetime.utcnow().isoformat(), 'message': 'testing'}
json.dumps(log_record)
# {"time": "2022-12-20T11:42:02.394702", "message": "testing"}

```

> this works, but this is far from ideal. Normally, our dictionary will contain the `datetime` object, not it’s string representation.
> 
> What we have to do is write custom code to replace non-JSON serializable objects in our dictionary with custom representations.
> 
> The simplest way is to specify a function that `dump` /`dumps` will call when it encounters something it cannot serialize

```python
def format_iso(dt):
    return dt.isoformat()

json.dumps(log_record, default=format_iso)
# {"time": "2022-12-20T11:43:12.487649", "message": "testing"}

log_record = {
    'time': datetime.utcnow(),
    'message': 'Testing...',
    'other': {'a', 'b', 'c'} # set尚未處理
}
json.dumps(log_record, default=format_iso)
# AttributeError

```

if else 判斷式，一個一個加上待處理的資料型態

```python
def custom_json_formatter(arg):
    if isinstance(arg, datetime):
        return arg.isoformat()
    elif isinstance(arg, set):
        return list(arg)

```

接著以萬年範例 class Person 做示範

程式碼略（原理都相同，只是換用 custom object 示範）

class Person 中的 datetime 因為已處理，不用再處理一次。

#### toJSON method

```python
class Person:
    def __init__ (self, name, age):
        self.name = name
        self.age = age
        self.create_dt = datetime.utcnow()
        
    def __repr__ (self):
        return f'Person(name={self.name}, age={self.age})'
    
    def toJSON(self):
        # return {
        # 'name': self.name,
        # 'age': self.age,
        # 'create_dt': self.create_dt
        # }
        return vars(self)

```

用一個較通用的方式，來處理更多不同的資料型態。

```python
def custom_json_formatter(arg):
    if isinstance(arg, datetime):
        return arg.isoformat()
    elif isinstance(arg, set):
        return list(arg)
    else:
        try:
            return arg.toJSON()
        except AttributeError:
            try:
                return vars(arg)
            except TypeError:
                return str(arg)

```

#### singledispatch

> re-write our custom json formatter using the generic single dispatch decorator.

```python
from functools import singledispatch

@singledispatch
def json_format(arg):
    print(arg)
    try:
        print('\ttrying to use toJSON...')
        return arg.toJSON()
    except AttributeError:
        print('\tfailed - trying to use vars...')
        try:
            return vars(arg)
        except TypeError:
            print('\tfailed - using string representation...')
            return str(arg)

```

然後再一一 `register` 不同的資料型態。

```python
@json_format.register(datetime)
def _(arg):
    return arg.isoformat()

```

```python
@json_format.register(set)
def _(arg):
    return list(arg)

```

最後用之前的 class Person 再示範一次。

```python
class Person:
    def __init__ (self, name, age):
        self.name = name
        self.age = age
        self.create_dt = datetime.utcnow()
        
    def __repr__ (self):
        return f'Person(name={self.name}, age={self.age})'
    
    def toJSON(self):
        return dict(name=self.name)
    # 比較用
        # 方法一：
        # return {
        # 'name': self.name,
        # 'age': self.age,
        # 'create_dt': self.create_dt
        # }
        # 方法二：
        # return vars(self)

```

* * *

### Custom JSON Encoding using JSONEncoder

#### JSONEncoder

> Encode the “standard” types, such as `str` , `int` , `float` , `list` , `dict` , etc.

| Python | JSON |
| --- | --- |
| dict | object {…} |
| list, tuple | array […] |
| str | string … |
| int, float | number |
| int or float Enums | number |
| bool | true or false |
| None | null |

> **[json — JSON encoder and decoder](https://docs.python.org/3/library/json.html#json.JSONEncoder)**
>
> Simple JSON decoder. | Source code: Lib/json/\_\_init\_\_.py JSON (JavaScript Object Notation), specified by RFC 7159(which obsoletes RFC 4627) and by ECMA-404, is a lightweight data interchange format inspired by JavaScript...

#### TypeError exception

> **non-supported objects**
> 
> We can actually extend this `JSONEncoder` class and override the **default** method.

```python
import json
from datetime import datetime

class CustomJSONEncoder(json.JSONEncoder):
    def default(self, arg):
        if isinstance(arg, datetime):
            return arg.isoformat()
        else:
            super().default(arg)

```
