Maya Python - Embed zip file into maya file?

Question

This was a suggestion from another stack thread that I'm finally getting back to. It was part of a discussion about how to embed tools into a maya file.

You can write the whole thing as a python package, zip it, then stuff the binary contents of the zip file into a fileInfo. When you need to code, look for it in the user's $MAYA_APP_DIR; if there's no zip, write the contents of the fileInfo to disk as a zip and then insert the zip into sys.path

Source discussions were: python copy scripts into script and Maya Python Create and Use Zipped Package?

So far the programming is going okay, but I think I hit a snag. When I attempt this:

with open("..directory/myZip.zip","rb") as file:
    cmds.fileInfo("myZip", file.read())

..and then I..

print cmds.fileInfo("myZip",q=1)

I get

[u'PK\003\004\024']

which is a bad translation of the first line of gibberish when reading the zip file as a text document.

How can I embed my zip file into my maya file as binary?

====================

Update: Maya doesn't like writing to the file as a direct read of the utf-8 encoded zip file. I found various methods of making it into an acceptable string that could be written, but the decoding back to the file didn't appear to work. I see now that Theodox's suggestion was to write it to binary and put that in the fileInfo node.

How can one encode, store and then decode to write to file later?

If I were to convert to binary using for instance:

' '.join(format(ord(x), 'b') for x in line)

What code would I need to turn that back into the original utf-8 zip information?

Answer 1

you can find related code here:

http://tech-artists.org/forum/showthread.php?4161-Maya-API-Singleton-Nodes&highlight=mayapersist

the relevant bit is

  import base64
  encoded = base64.b64encode(value)
  decoded = base64.b64decode(encoded)

basically it's the same idea, except using the base64 module instead of binascii. Any method of converting an arbitary character stream to an ascii-safe representation will work fine, as long as you use a reversable method: the potential problem you need to watch out for is a character in the data block that looks to maya like a close quote - an open quote int he fileInfo will be messy in an MA file.

This example uses YAML to do arbitrary key-value pairs but that part is irrelevant to storing the binary stuff. I've used this technique for fairly large data (up to 640k if i recall) but I don't know if it has an upper limit in terms of what you can stash in Maya

Answer 2

Found the answer. Great script on stack overflow. I had to encode to 'string_escape', which is something I found while trying to figure out the whole characters situation. But anyways, you open the zip, encode to 'string_escape', write it to fileInfo and then before fetching to write it back to zip, you decode it back.

Convert binary to ASCII and vice versa

import maya.cmds as cmds
import binascii

def text_to_bits(text, encoding='utf-8', errors='surrogatepass'):
    bits = bin(int(binascii.hexlify(text.encode(encoding, errors)), 16))[2:]
    return bits.zfill(8 * ((len(bits) + 7) // 8))

def text_from_bits(bits, encoding='utf-8', errors='surrogatepass'):
    n = int(bits, 2)
    return int2bytes(n).decode(encoding, errors)

def int2bytes(i):
    hex_string = '%x' % i
    n = len(hex_string)
    return binascii.unhexlify(hex_string.zfill(n + (n & 1)))

And then you can

with open("..\maya\scripts/test.zip","rb") as thing:
    texty = text_to_bits(thing.read().encode('string_escape'))
    cmds.fileInfo("binaryZip",texty)

...later

with open("..\maya\scripts/test_2.zip","wb") as thing:
    texty = cmds.fileInfo("binaryZip",q=1)
    thing.write( text_from_bits( texty ).decode('string_escape') )

and this appears to work.. so far..

Answer 3

Figured I'd post the final product for anyone wanting to undertake this approach. I tried to do some corruption checking, so that a bad zip doesn't get passed between machines. That's what all the check hashing is for.

def writeTimeFull(tl):
    import TimeFull
    #reload(TimeFull)
    with open(TimeFull.__file__.replace(".pyc",".py"),"r") as file:
        cmds.scriptNode( tl.scriptConnection[1][0], e=1, bs=file.read() )
    cmds.expression("spark_timeliner_activator",
                    e=1,s='if (Spark_Timeliner.ShowTimeliner == 1)\n'
                          '{\n'
                          '\tsetAttr Spark_Timeliner.ShowTimeliner 0;\n'
                          '\tpython \"Timeliner.InitTimeliner()\";\n'
                          '}',
                    o="Spark_Timeliner",ae=1,uc=all)

def checkHash(zipPath,hash1,hash2,hash3):
    check = False
    hashes = [hash1,hash2,hash3]
    for ii, hash in enumerate(hashes):
        if hash == hashes[(ii+1)%3]:
            hashes[(ii+2)%3] = hashes[ii]
            check = True
    if check:
        if md5(zipPath) == hashes[0]:
            return [zipPath,hashes[0],hashes[1],hashes[2]]
    else:
        cmds.warning("Hash checks and/or zip are corrupted. Attain toolbox_fix.zip, put it in scripts folder and restart.")
        return []

#this writes the zip file to the local users space
def saveOutZip(filename):
    if os.path.isfile(filename):
        if not os.path.isfile(filename.replace('_pkg','_'+__version__)):
            os.rename(filename,filename.replace('_pkg','_'+__version__))
    with open(filename,"w") as zipFile:
        zipInfo = cmds.fileInfo("zipInfo1",q=1)[0]
        zipHash_1 = cmds.fileInfo("zipHash1",q=1)[0]
        zipHash_2 = cmds.fileInfo("zipHash2",q=1)[0]
        zipHash_3 = cmds.fileInfo("zipHash3",q=1)[0]
        zipFile.write( base64.b64decode(zipInfo) )
        if checkHash(filename,zipHash_1,zipHash_2,zipHash_3):
            cmds.fileInfo("zipInfo2",zipInfo)
            return filename
    with open(filename,"w") as zipFile:
        zipInfo = cmds.fileInfo("zipInfo2",q=1)[0]
        zipHash_1 = cmds.fileInfo("zipHash1",q=1)[0]
        zipHash_2 = cmds.fileInfo("zipHash2",q=1)[0]
        zipHash_3 = cmds.fileInfo("zipHash3",q=1)[0]
        zipFile.write( base64.b64decode(zipInfo) )
        if checkHash(filename,zipHash_1,zipHash_2,zipHash_3):
            cmds.fileInfo("zipInfo1",zipInfo)
            return filename
    return False

#this writes the local zip to this file
def loadInZip(filename):
    zipResults = []
    for ii in range(0,10):
        with open(filename,"r") as theRead:
            zipResults.append([base64.b64encode(theRead.read())]+checkHash(filename,md5(filename),md5(filename),md5(filename)))
        if ii>0 and zipResults[ii]==zipResults[ii-1]:
            cmds.fileInfo("zipInfo1",zipResults[ii][0])
            cmds.fileInfo("zipInfo2",zipResults[ii-1][0])
            cmds.fileInfo("zipHash1",zipResults[ii][2])
            cmds.fileInfo("zipHash2",zipResults[ii][3])
            cmds.fileInfo("zipHash3",zipResults[ii][4])
            return True

#file check
#http://stackoverflow.com/questions/3431825/generating-a-md5-checksum-of-a-file
def md5(fname):
    import hashlib
    hash = hashlib.md5()
    with open(fname, "rb") as f:
        for chunk in iter(lambda: f.read(4096), b""):
            hash.update(chunk)
    return hash.hexdigest()

filename = path+'/toolbox_pkg.zip'
zipPaths = [path+'/toolbox_update.zip',
            path+'/toolbox_fix.zip',
            path+'/toolbox_'+__version__+'.zip',
            filename]
zipPaths_exist = [os.path.isfile(zipPath) for zipPath in zipPaths ]

if any(zipPaths_exist[:2]):
    if zipPaths_exist[0]:
        cmds.warning('Timeliner update present. Forcing file to update version')
        if zipPaths_exist[2]:
            os.remove(zipPaths[3])
        elif os.path.isfile(zipPaths[3]):
            os.rename(zipPaths[3], zipPaths[2])
        os.rename(zipPaths[0],zipPaths[3])
        if zipPaths_exist[1]:
            os.remove(zipPaths[1])
    else:
        cmds.warning('Timeliner fix present. Replacing file to the fix version.')
        if os.path.isfile(zipPaths[3]):
            os.remove(zipPaths[3])
        os.rename(zipPaths[1],zipPaths[3])
    loadInZip(filename)

if not cmds.fileInfo("zipInfo1",q=1) and not cmds.fileInfo("zipInfo2",q=1):
    loadInZip(filename)

if not os.path.isfile(filename):
    saveOutZip(filename)

sys.path.append(filename)
import Timeliner
Timeliner.InitTimeliner(theVers=__version__)

if not any(zipPaths[:2]):
    if __version__ > Timeliner.__version__:
        cmds.warning('Saving out newer version of timeliner to local machine. Restart Maya to access latest version.')
        saveOutZip(filename)
    elif __version__ < Timeliner.__version__:
        cmds.warning('Timeliner on machine is newer than file version. Saving machine version over timeliner in file.')
        loadInZip(filename)
        __version__ = Timeliner.__version__

if __name__ != "__main__":
    tl = getTimeliner()
    writeTimeFull(tl)