How to save a large array in txt file in python

Question

I have 2D array:

import numpy as np

output = np.array([1,1,6])*np.arange(6)[:,None]+1

output
Out[32]: 
array([[ 1,  1,  1],
       [ 2,  2,  7],
       [ 3,  3, 13],
       [ 4,  4, 19],
       [ 5,  5, 25],
       [ 6,  6, 31]])

I tried to use np.savetxt('file1.txt', output, fmt='%10d') i have got the result in one line only

How can I save it in txt file simillar to :

3 separate columns, each column has name (x,y,z)

Please note: the original array too large (40000000 rows and 3 columns), I am using Python 3.6 I have tried the solutions in here and here but, it does not work with me

40mio*3*3 characters for the data and spacing are roughly 350mb assuming 1 byter per character - thats not a too big file. — Patrick Artner
– Patrick Artner, Commented Dec 31, 2017 at 21:38
@PatrickArtner: but if we use text, it uses separators (tab and new line), and some data (e.g. floats) can use ~10-20 digits. — willeM_ Van Onsem
– willeM_ Van Onsem, Commented Dec 31, 2017 at 21:39
@WillemVanOnsem sure, his demodata are ints in range below 99 though and I rounded up ;) just trying to get a feel about his problem — Patrick Artner
– Patrick Artner, Commented Dec 31, 2017 at 21:40

Patrick Artner · Accepted Answer · 2017-12-31 22:20:31Z

2

Noor, let me guess - you are using windows notepad to view the file?

I use Notepad++ which is smart enough to understand Unix-style-Lineendings which are used (by default) when creating files by np.savetxt() even when operated under windows.

You might want to explicitly specify newline="\r\n" when calling savetxt.

np.savetxt('file1.txt', output, fmt='%10d' ,header= "       x          y          z", newline="\r\n")

Doku: https://docs.scipy.org/doc/numpy-1.13.0/reference/generated/numpy.savetxt.html

I am not sure about your data, but this:

import numpy as np

output = np.array([1,1,6])*np.arange(60)[:,None]+1

print(output)


np.savetxt('file1.txt', output, fmt='%10d' ,header= "       x          y          z")

Produces this output:

#        x          y          z
         1          1          1
         2          2          7
         3          3         13 
       === snipped a few lines ===
        58         58        343
        59         59        349
        60         60        355

for me.

for np.arange(1000000) its about 32MB big and similarly formatted...
for np.arange(10000000) its about 322MB big and similarly formatted...

willem-van-onsem 1+Gb was far closer.

I did not account for the spacing of fixed 10 chars per number, my bad.

edited Dec 31, 2017 at 22:20

answered Dec 31, 2017 at 22:02

Patrick Artner

51.9k10 gold badges50 silver badges79 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Noor Over a year ago

Thanks, Patrick Artner, you are right, Notepad++, and EmEditor, both of them display the array in columns and rows

Patrick Artner Over a year ago

@Noor - specify the newline and windows style one are used- wich adds 40.000.000 bytes to your overall file size ;)

Collectives™ on Stack Overflow

How to save a large array in txt file in python

1 Answer 1

2 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related