Skip to main content

Reducing NumPy memory usage with lossless compression.


If you’re running into memory issues because your NumPy arrays are too large, one of the basic approaches to reducing memory usage is compression. By changing how you represent your data, you can reduce memory usage and shrink your array’s footprint—often without changing the bulk of your code.

In this article we’ll cover:

    * Reducing memory usage via smaller dtypes.
    * Sparse arrays.
    * Some situations where these solutions won’t work.

Comments

Post a Comment

Popular posts from this blog

Fixing Unix/Linux/POSIX Filenames

Traditionally, Unix/Linux/POSIX filenames can be almost any sequence of bytes, and their meaning is unassigned. The only real rules are that "/" is always the directory separator, and that filenames can't contain byte 0 (because this is the terminator). Although this is flexible, this creates many unnecessary problems. In particular, this lack of limitations makes it unnecessarily difficult to write correct programs (enabling many security flaws), makes it impossible to consistently and accurately display filenames, causes portability problems, and confuses users. more ....
Post a Comment
Read more

Multi-Boot Disk for Machines With AMD Opteron Processors

This article presents step-by-step procedures for loading the Solaris 10 OS on x86 platforms, and one or two 64-bit Linux operating systems, on machines based on 64-bit AMD Opteron processors. Installations were done on generic Opteron-based workstations and confirmed on a Sun Fire V20z server and Sun Java Workstation W1100z and W2100z workstations.
Post a Comment
Read more