Python: writing large array of arrays to text file

Question

I'm new to Python and I have a solution for this but it seems slow and silly, so I wondered if there is a better way?

Say I have a matrix defined like this:

mat = [['hello']*4 for x in xrange(3)]

I am using this function to write it to file:

def writeMat(mat, outfile):
  with open(outfile, "w") as f:
    for item in mat:
      f.writelines(str(item).replace('[','').replace(',','').replace('\'','').replace(']','\n'))

writeMat(mat, "temp.txt")

which gives a text file that looks like:

hello hello hello hello
hello hello hello hello
hello hello hello hello

The files that I am dealing with are very large. The savetxt function in numpy would be great, but I don't want to store this as a numpy array because while the majority of the matrix is comprised of single character elements, the first few columns will be many characters in length, and it seems to me (correct me if I am wrong) this would mean the whole matrix would use much more memory than is necessary because every element in the matrix will be the size of the largest element.

The internal implementation of numpy is smarter than you think. ;-) — Keith
– Keith, Commented Jul 21, 2012 at 5:39

Joel Cornett · Accepted Answer · 2012-07-21 05:23:53Z

2

If I understand your question correctly, you could do:

f.writelines(' '.join(row) + '\n' for row in mat)

or

f.write('\n'.join(' '.join(row) for row in mat))

The first one has the advantage of being a generator expression that only makes a concatenated string copy of the currentline

And if your matrix entries are not strings, you could do:

f.writelines(' '.join(str(elem) for elem in row) + '\n' for row in mat)

EDIT

It appears that the file.writelines() method evaluates the entire generator expression before writing it to the file. So the following would minimize your memory consumption:

for row in mat:
    f.write(' '.join(row) + '\n')

edited Jul 21, 2012 at 5:23

answered Jul 21, 2012 at 5:10

Joel Cornett

24.8k9 gold badges69 silver badges90 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

explodecomputer Over a year ago

This is much more straightforward. Learning learning.. thanks!

jfs · Accepted Answer · 2012-07-21 05:30:40Z

1

You could use csv module:

import csv

with open(outfile, 'wb') as f:
     csv.writer(f, delimiter=' ').writerows(mat)

answered Jul 21, 2012 at 5:30

jfs

417k210 gold badges1k silver badges1.7k bronze badges

1 Comment

explodecomputer Over a year ago

Thanks for the suggestion, this will be useful for me in the future

Collectives™ on Stack Overflow

Python: writing large array of arrays to text file

2 Answers 2

1 Comment

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related