How To Read From A Text File Compressed With 7z?

I would like to read (in Python 2.7), line by line, from a csv (text) file, which is 7z compressed. I don't want to decompress the entire (large) file, but to stream the lines. I t

Solution 1:

This will allow you to iterate the lines. It's partially derived from some code I found in an answer to another question.

At this point in time (pylzma-0.5.0) the py7zlib module doesn't implement an API that would allow archive members to be read as a stream of bytes or characters — its ArchiveFile class only provides a read() function that decompresses and returns the uncompressed data in a member all at once. Given that, about the best that can be done is return bytes or lines iteratively via a Python generator using that as a buffer.

The following does the latter, but may not help if the problem is the archive member file itself is huge.

The code below should work in Python 3.x as well as 2.7.

import io
import os
import py7zlib


classSevenZFileError(py7zlib.ArchiveError):
    passclassSevenZFile(object):
    @classmethoddefis_7zfile(cls, filepath):
        """ Determine if filepath points to a valid 7z archive. """
        is7z = False
        fp = Nonetry:
            fp = open(filepath, 'rb')
            archive = py7zlib.Archive7z(fp)
            _ = len(archive.getnames())
            is7z = Truefinally:
            if fp: fp.close()
        return is7z

    def__init__(self, filepath):
        fp = open(filepath, 'rb')
        self.filepath = filepath
        self.archive = py7zlib.Archive7z(fp)

    def__contains__(self, name):
        return name in self.archive.getnames()

    defreadlines(self, name, newline=''):
        r""" Iterator of lines from named archive member.

        `newline` controls how line endings are handled.

        It can be None, '', '\n', '\r', and '\r\n' and works the same way as it does
        in StringIO. Note however that the default value is different and is to enable
        universal newlines mode, but line endings are returned untranslated.
        """
        archivefile = self.archive.getmember(name)
        ifnot archivefile:
            raise SevenZFileError('archive member %r not found in %r' %
                                  (name, self.filepath))

        # Decompress entire member and return its contents iteratively.
        data = archivefile.read().decode()
        for line in io.StringIO(data, newline=newline):
            yield line


if __name__ == '__main__':

    import csv

    if SevenZFile.is_7zfile('testing.csv.7z'):
        sevenZfile = SevenZFile('testing.csv.7z')

        if'testing.csv'notin sevenZfile:
            print('testing.csv is not a member of testing.csv.7z')
        else:
            reader = csv.reader(sevenZfile.readlines('testing.csv'))
            for row in reader:
                print(', '.join(row))

Solution 2:

If you were using Python 3.3+, you might be able to do this using the lzma module which was added to the standard library in that version.

See: lzmaExamples

Solution 3:

If you can use python 3, there is a useful library, py7zr, which supports partially 7zip decompression as below:

import py7zr
import re
filter_pattern = re.compile(r'<your/target/file_and_directories/regex/expression>')
with SevenZipFile('archive.7z', 'r') as archive:
    allfiles = archive.getnames()
    selective_files = [f if filter_pattern.match(f) for f in allfiles]
    archive.extract(targets=selective_files)