fickle/PKG-INFO at main · roboflow/fickle · GitHub

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
Metadata-Version: 2.1
Name: fickle
Version: 0.2.2
Summary: load pickled data as safely as possible
Home-page: UNKNOWN
Author: Eduard Christian Dumitrescu
Author-email: eduard.c.dumitrescu@gmail.com
License: MIT
Description: # Fickle: Firewalled Pickle

        People abuse pickle. Especially researchers. Pickle is [not secure](https://docs.python.org/3/library/pickle.html). Published datasets and ML training weights are often distributed as pickle files (or formats which use pickle files, such as [PyTorch checkpoint.ckpt files](https://pytorch.org/tutorials/beginner/saving_loading_models.html)). Sometimes it is the only format that they are available in.

        ## Examples

        Loading basic types is easy:

        ```python
        >>> from fickle import DefaultFirewall
        >>> import pickle
        >>>
        >>> my_picked_data = pickle.dumps({"list": [1, 2, "three", b"four"]})
        >>>
        >>> firewall = DefaultFirewall()
        >>> firewall.loads(my_picked_data)
        {'list': [1, 2, 'three', b'four']}
        ```

        Safely loading PyTorch checkpoint files into numpy arrays is just as easy:

        ```python
        >>> from fickle.ext.pytorch import fake_torch_load_zipped
        >>> from zipfile import ZipFile
        >>>
        >>> zf = ZipFile("/path/to/sd-v1-4.ckpt")
        >>> ckpt = fake_torch_load_zipped(zf)
        >>> tensor = ckpt["state_dict"]["model.diffusion_model.output_blocks.3.1.norm.weight"]
        >>> tensor.array
        array([0.39097363, 0.3898967 , 0.35191917, ..., 0.41924757, 0.4031702 ,
               0.37156993], dtype=float32)
        ```

        You can, optionally, even use `marshmallow` for validation!

        ## Alternatives

        - [picklemagic](https://github.com/CensoredUsername/picklemagic)
        - [pikara](https://pypi.org/project/pikara)

        |                                                | fickle | picklemagic | pikara |
        |------------------------------------------------|--------|-------------|--------|
        | Does not rely on `pickle._Unpickler`?          | ✅      | ❌           | ✅      |
        | Uses `pickletools.genops`                      | yes    | no          | yes    |
        | Can load without executing?                    | ✅      | ✅           | ?      |
        | Forbid importing arbitrary objects?            | ✅      | ✅           | ?      |
        | Forbid calling `list.append`/`set.add`/etc?    | ✅      | ❌           | ?      |
        | Forbid calling all methods by default?         | ✅      | ❌           | ?      |
        | Can create dangerous circular structures?      | ✅      | ✅           | ?      |
        | Safe against billion laughs DoS attack?        | ?      | ?           | ?      |
        | Full support for all pickle opcodes?           | ❌      | ✅           | ?      |
        | Has unit tests?                                | ✅      | ❌           | ✅      |
        | Stable API?                                    | ❌      | ✅           | ✅      |

Platform: UNKNOWN
Classifier: Programming Language :: Python :: 3
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Description-Content-Type: text/markdown