Skip to main content

Reducing NumPy memory usage with lossless compression.


If you’re running into memory issues because your NumPy arrays are too large, one of the basic approaches to reducing memory usage is compression. By changing how you represent your data, you can reduce memory usage and shrink your array’s footprint—often without changing the bulk of your code.

In this article we’ll cover:

    * Reducing memory usage via smaller dtypes.
    * Sparse arrays.
    * Some situations where these solutions won’t work.

Comments

Post a Comment

Popular posts from this blog

Fixing Unix/Linux/POSIX Filenames

Traditionally, Unix/Linux/POSIX filenames can be almost any sequence of bytes, and their meaning is unassigned. The only real rules are that "/" is always the directory separator, and that filenames can't contain byte 0 (because this is the terminator). Although this is flexible, this creates many unnecessary problems. In particular, this lack of limitations makes it unnecessarily difficult to write correct programs (enabling many security flaws), makes it impossible to consistently and accurately display filenames, causes portability problems, and confuses users. more ....
Post a Comment
Read more

Debian and Windows Shared Printing.

Debian GNU/Linux ( http://www.debian.org ) is the premier volunteer-supported Linux distribution. Unfortunately, setting up printers in Debian can be difficult. Also, simple step-by-step instructions for sharing printers between Windows and Linux using the latest tools are hard to find. This HOWTO was written to address both problems. This HOWTO will demonstrate how to use command-line tools to configure your Debian system for printing. It will explain how to send documents from Linux to Windows printers and how to share Linux printers with Windows PCs. Some troubleshooting examples are also given.
Post a Comment
Read more

Backing up MySQL data.

Backing up files and directories is relatively easy; databases, however, have some special quirks that you need to address. Our examples use MySQL, but the same principles apply to PostgreSQL and other relational databases. This article is excerpted from O'Reilly's recently published book Linux System Administration .
Post a Comment
Read more