1

I've been trying to concatenate two arrays (string array and float array) and then write the final array to a CSV file. However, I've been facing a lot of problems to solve it. I already tried a lot of tutorials in the internet about this topic, but no one worked properly for me.

The code that I`m writing use python to automate a serie of tasks in ArcGIS. After the code run and get all the mean values, I need to join the mean value from the raster file together with two strings.

I`ve already tried to join all the informations that I need in one single array, but I face problems when I try to save it to a CSV file.

The code is the following.

#Interpolation
        arcpy.gp.Idw_sa(out_Layer_shp, i_dia, arcpy.env.workspace + "\\Raster\\" + out_Layer_tif + ".tif", tamanho_celula, potencia_idw, "VARIABLE 12 12", "") 


        #Deleting the shape file
        arcpy.Delete_management(out_Layer_shp, "") 



        lista_final = np.array([], dtype = object)  #Final list
        contador = 0 #Counter 


        for mascara in bacias: #Importing shapefiles to clip the interpolated raster
            importar_camada = arcpy.env.workspace + "\\Shapefile\\Bacias\\" + mascara + ".shp"  
            arcpy.MakeFeatureLayer_management(importar_camada, mascara)  



            #Variable to set the name of the layers that will be created and then clipped
            camada_para_recortar = out_Layer_tif + ".tif"
            camada_resultante = out_Layer + "_recortada"
            nome_do_raster = camada_resultante + "_" + mascara + ".tif" 

            #Code to interpolate
            arcpy.Clip_management(camada_para_recortar, "", arcpy.env.workspace  + "\\Raster\\Recortes\\" + camada_resultante + "_" + mascara + ".tif", mascara, "-3,402823e+038", "ClippingGeometry", "NO_MAINTAIN_EXTENT")

            #Getting the mean value from the raster clipped
            media = arcpy.GetRasterProperties_management (nome_do_raster, "MEAN", "") 


            lista_strings = np.array([out_Layer, mascara]) #array string
            lista_medias = np.array([media]) #float string 

            arquivo_com_as_medias = "medias 01" #Name of the file to save the means values

            lista_numpy_temporaria = np.array([out_Layer, mascara, media], dtype = object) #Temporary list to save the data before
                                                                                                                        #it be added to the final array
            #lista_numpy_temporaria = lista_numpy_temporaria.reshape(1,3)

            lista_final = np.concatenate((lista_final, lista_numpy_temporaria)) #Concatenating the two arrays

            contador = contador + 1 #Incrementing the counter

        lista_final = lista_final.reshape(contador,3)    #reshape the final array   


        np.savetxt(arcpy.env.workspace  + "\\Dados\\Exportados\\" + arquivo_com_as_medias + ".csv", lista_final, fmt="%10s %10.3f", delimiter=";", header = "#1, #2")




        #Attempt to write the arrays based in a tutorial that I found, but it didn't work
        """         
        ab = np.zeros(lista_numpy_temporaria.size, dtype=[('var1', 'U6'), ('var2', float)])
        ab['var1'] = lista_strings
        ab['var2'] = lista_medias

        np.savetxt(arcpy.env.workspace  + "\\Dados\\Exportados\\" + arquivo_com_as_medias + ".csv", ab, fmt="%10s %10.3f")
        """


        print(lista_final)

The error message that I get from this code is the following

Runtime error Traceback (most recent call last): File "", line 50, in File "C:\Python27\ArcGIS10.3\lib\site-packages\numpy\lib\npyio.py", line 1031, in savetxt raise error ValueError: fmt has wrong number of % formats: %10s %10.3f

The code have a lot of comments because of the many attempts that I've been making.

What I aiming to get is a CSV file with 3 columns and a lot of rows, the number of rows will vary depending the amount of data been interpolated. One example would be following:

RASTER FILE       SUB_BASIN       MEAN_VALUE 
out_Layer         mascara         99.99

Thank you.

Edit 1

The new message error after the solution suggested is the following:

Runtime error Traceback (most recent call last): File "", line 210, in File "C:\Python27\ArcGIS10.3\lib\site-packages\numpy\lib\npyio.py", line 1047, in savetxt fh.write(asbytes(format % tuple(row) + newline)) TypeError: float argument required, not Result

The solution that I've found was to change the format from fmt="%10s %10s %10.3f" to this

fmt="%10s %10s %10s"

Although it is not the best solution, since it return numbers as strings.

The full code that I'm using is the following.

import arcpy
import numpy  as np

from arcpy import sa 
from arcpy.sa import *
from calendar import monthrange


# Set environment settings
arcpy.env.workspace = "C:\\Projetos\\ArcGIS\\Teste9"  

arcpy.env.overwriteOutput = True 


#get the map document
mxd = arcpy.mapping.MapDocument("CURRENT")


#get the data frame
df = arcpy.mapping.ListDataFrames(mxd,"la*")[0]





#Months and years to interpolate
mes = ["2"] #Months
ano = ["1994"] #Years

#Days to interpolate based in the month lenght
coluna_interpolada_28 = ["D1", "D2", "D3", "D4", "D5", "D6", "D7", "D8", "D9", "D10", "D11", "D12", "D13", "D14", "D15", "D16", "D17", "D18", "D19", "D20", "D21", "D22", "D23", "D24", "D25", "D26", "D27", "D28"]  
coluna_interpolada_29 = ["D1", "D2", "D3", "D4", "D5", "D6", "D7", "D8", "D9", "D10", "D11", "D12", "D13", "D14", "D15", "D16", "D17", "D18", "D19", "D20", "D21", "D22", "D23", "D24", "D25", "D26", "D27", "D28", "D29"]
coluna_interpolada_30 = ["D1", "D2", "D3", "D4", "D5", "D6", "D7", "D8", "D9", "D10", "D11", "D12", "D13", "D14", "D15", "D16", "D17", "D18", "D19", "D20", "D21", "D22", "D23", "D24", "D25", "D26", "D27", "D28", "D29", "D30"]
coluna_interpolada_31 = ["D1", "D2", "D3", "D4", "D5", "D6", "D7", "D8", "D9", "D10", "D11", "D12", "D13", "D14", "D15", "D16", "D17", "D18", "D19", "D20", "D21", "D22", "D23", "D24", "D25", "D26", "D27", "D28", "D29", "D30", "D31"]



#Interpolation extent
arcpy.env.extent = arcpy.env.workspace + "\\Shapefile\\" + "PB.shp"



#Final list
lista_final = np.array([], dtype = object) 


#Counter that is going to be used to reshape the arrays
contador = 0 


#Loop to go through the time series
for i_ano in ano: #For loop with the years

    for i_mes in mes: #For loop with the months

        #Month Range
        quantidade_dias = monthrange(int(i_ano), int(i_mes))

        #If clauses to define which columns it is goin to interpolate
        if quantidade_dias == (1, 28):
            coluna_interpolada = coluna_interpolada_28
        elif quantidade_dias == (1, 29):
            coluna_interpolada = coluna_interpolada_29
        elif quantidade_dias == (1, 30):
            coluna_interpolada = coluna_interpolada_30
        else:
            coluna_interpolada = coluna_interpolada_31

        #For loop with the days
        for i_dia in coluna_interpolada:

            tabela = i_mes + "_" + i_ano   #Exemplo "2_1994"

            in_Table = arcpy.env.workspace + "\\Dados\\" + tabela + ".csv" #Exemplo "2_1994.csv"
            x_coords = "LONG"
            y_coords = "LAT"
            z_coords = "POINT_Z"
            out_Layer = "espacializacao" + "_" + tabela + "_" + i_dia #nome da camada "lyr" que vai ser criada   
                                                                                        #NOME DO ARQUIVO QUE VAI SER SALVO. Exemplo "espacializacao_2_1994_D2"
            out_Layer_shp = out_Layer + "_shp" #nome da camada "shp" que vai ser criada
            out_Layer_tif = out_Layer + "_tif"
            tamanho_celula = "0,10" #precisar por as aspas, apesar de ser um número
            potencia_idw = "2" #precisa por as aspas, apesar de ser um número
            raio_de_busca = RadiusVariable(12, 12) #Critério para fazer a interpolação, raio variando até 12 quilômetros até conseguir englobar 12 pontos




            # Set the spatial reference
            spRef = arcpy.SpatialReference("WGS 1984")

            #Create event layer
            arcpy.MakeXYEventLayer_management(in_Table, x_coords, y_coords, out_Layer, spRef, "")


            #Exporting event layer as shapefile
            arcpy.FeatureToPoint_management(out_Layer, arcpy.env.workspace + "\\" + "Shapefile\\Exportados\\" + out_Layer_shp + ".shp","")





            #Layer that is going to be deleted "lyr"
            lyr = arcpy.mapping.ListLayers(mxd, "espacializacao",df)



            #Deleting the layer
            for df in arcpy.mapping.ListDataFrames(mxd):
                 for lyr in arcpy.mapping.ListLayers(mxd, "", df): #O parâmetro que não foi passado foi o WildCard, não precisa
                     if lyr.name == out_Layer:
                         arcpy.mapping.RemoveLayer(df, lyr) #Removendo a camada da paleta lateral





            #Some variables to define some parameters to the software
            camada_editando = out_Layer_shp
            coluna_criada = "Media"
            tipo_campo = "FLOAT"
            precisao_campo = ""
            precisao_decimais = ""
            comprimento_campo = 50 #Tamanho qualquer suposto
            arcpy.AddField_management(camada_editando, coluna_criada, "LONG", precisao_campo,
                                          precisao_decimais, comprimento_campo, "", "NULLABLE",
                                          "NON_REQUIRED", "")   
            bacias = ["Abiai", "Camaratuba", "Curimatau", "Gramame", "Guaju", "Jacu", "Mamanguape", "Miriri", "Paraiba", "Piranhas", "Trairi"]          




            #Code to interpolate
            arcpy.gp.Idw_sa(out_Layer_shp, i_dia, arcpy.env.workspace + "\\Raster\\" + out_Layer_tif + ".tif", tamanho_celula, potencia_idw, "VARIABLE 12 12", "") 


            #Deleting shapefile
            arcpy.Delete_management(out_Layer_shp, "")  






            #For loop to clip the raster file using the shapefiles
            for mascara in bacias:
                importar_camada = arcpy.env.workspace + "\\Shapefile\\Bacias\\" + mascara + ".shp"  #Importing shapefile to clip
                arcpy.MakeFeatureLayer_management(importar_camada, mascara)  



                #Some variables defining some parameters to using in the clip function
                camada_para_recortar = out_Layer_tif + ".tif"
                camada_resultante = out_Layer + "_recortada"
                nome_do_raster = camada_resultante + "_" + mascara + ".tif" 

                #Function to clip the raster file
                arcpy.Clip_management(camada_para_recortar, "", arcpy.env.workspace  + "\\Raster\\Recortes\\" + camada_resultante + "_" + mascara + ".tif", mascara, "-3,402823e+038", "ClippingGeometry", "NO_MAINTAIN_EXTENT")
                media = arcpy.GetRasterProperties_management (nome_do_raster, "MEAN", "")


                lista_strings = np.array([out_Layer, mascara])
                lista_medias = np.array([media])


                #Name of the file to save the means values
                arquivo_com_as_medias = "medias 01" 


                lista_numpy_temporaria = np.append(lista_strings, lista_medias)
                lista_final = np.concatenate((lista_final, lista_numpy_temporaria))


                #Deleting the raster clipped
                arcpy.Delete_management(nome_do_raster, "") 

                #Counter
                contador = contador + 1 

                print(lista_final)

            #Reshaping the file 
            lista_final = lista_final.reshape(contador,3)       

            print(lista_final)

            #Saving the arrays to a CSV file
            np.savetxt(arcpy.env.workspace  + "\\Dados\\Exportados\\" + arquivo_com_as_medias + ".csv", lista_final, fmt="%10s %10s %10s", delimiter=";", header = "")




            #Deleting the original raster
            arcpy.Delete_management(out_Layer_tif + ".tif", "")

Using this code above, the "print(lista_final)" return the following:

    >>> print(lista_final)
[[u'espacializacao_2_1994_D1' u'Abiai' u'111,81740045547']
 [u'espacializacao_2_1994_D1' u'Camaratuba' u'328,11316935221']
 [u'espacializacao_2_1994_D1' u'Curimatau' u'273,3234489704']
 [u'espacializacao_2_1994_D1' u'Gramame' u'223,45285224915']
 [u'espacializacao_2_1994_D1' u'Guaju' u'393,62130737305']
 [u'espacializacao_2_1994_D1' u'Jacu' u'312,91506958008']
 [u'espacializacao_2_1994_D1' u'Mamanguape' u'289,06595204671']
 [u'espacializacao_2_1994_D1' u'Miriri' u'564,86507415771']
 [u'espacializacao_2_1994_D1' u'Paraiba' u'330,80016106998']
 [u'espacializacao_2_1994_D1' u'Piranhas' u'328,95194289264']
 [u'espacializacao_2_1994_D1' u'Trairi' u'333,04579162598']]

Another thing that I was thinking was, would this be the best approach to write these outputs in a CSV file? Because I already saw some tutorials using the "CSV module" to write directly to CSV files. However, I already tried it without success.

I am saying this because this code is going to be repeated a lot of times, something like 19.000 times. Because I am using it to interpolate daily rainfall data from a 54 years time series. So, I guess that an array of this size is not a good approach to solve this problem.

Thank you, once more.

Edit 2

I tried what you suggested, but it's still not working. The changes I did were

#First attempt 
one = np.array([out_Layer], dtype = object)  
two = np.array([mascara], dtype = object)    
three = np.array([media])      

rarr = np.rec.fromarrays([one, two, three])   

arquivo_com_as_medias = "medias 01" #File with the means values 
csv_directory = arcpy.env.workspace  + "\\Dados\\Exportados\\" + arquivo_com_as_medias + ".csv" 

np.savetxt(csv_directory, rarr, fmt=['%s', '%s', '%f'], delimiter=";")

And the error message it's showing now is the following

Runtime error Traceback (most recent call last): File "", line 176, in File "C:\Python27\ArcGIS10.3\lib\site-packages\numpy\core\records.py", line 560, in fromarrays raise ValueError("array-shape mismatch in array %d" % k) ValueError: array-shape mismatch in array 2

The code in the line 176 is

rarr = np.rec.fromarrays([one, two, three])
Improve this question

2 Answers 2

Reset to default
1

At least as far as the error you're getting goes, it's like it says: you're trying to output 3 columns, but are only providing two formats. If you change your fmt arg to something like fmt=['%10s', '%10s', '%10.3f'], then np.savetxt should work.

It's possible you'll run into other errors, such as those relating to the format/contents of the array you're trying to save out. In order to sidestep any such potential errors, you may also want to temporarily add a line to your script like:

print(lista_final[:100])

right before the np.savetxt line. That way, you can make sure that the contents of lista_final (at least in the first hundred rows) are actually what you expect them to be before you try to process the array through savetxt.

Edit

When you call np.concatenate((lista_final, lista_numpy_temporaria)) near the end of your code, you end up forcing all 3 of your columns into a single standard Numpy array. A standard array can only have a single dtype, so everything, including the floats in the last column, end up getting converted to strings. You can avoid this by instead joining your columns into a record array (which can have a different dtype for every column). Say that the three column arrays that you're trying to bring together at the end of your script are like this:

one = np.array(["u'espacializacao_2_1994_D1'", "u'espacializacao_2_1994_D1'",
       "u'espacializacao_2_1994_D1'", "u'espacializacao_2_1994_D1'",
       "u'espacializacao_2_1994_D1'", "u'espacializacao_2_1994_D1'",
       "u'espacializacao_2_1994_D1'", "u'espacializacao_2_1994_D1'",
       "u'espacializacao_2_1994_D1'", "u'espacializacao_2_1994_D1'",
       "u'espacializacao_2_1994_D1'"], dtype=object)

two = np.array(["u'Abiai'", "u'Camaratuba'", "u'Curimatau'", "u'Gramame'",
       "u'Guaju'", "u'Jacu'", "u'Mamanguape'", "u'Miriri'", "u'Paraiba'",
       "u'Piranhas'", "u'Trairi'"], dtype=object)

three = np.array([111.81740046, 328.11316935, 273.32344897, 223.45285225,
       393.62130737, 312.91506958, 289.06595205, 564.86507416,
       330.80016107, 328.95194289, 333.04579163])

You would then join them together in a single record array and then write them out to a csv file like so:

rarr = np.rec.fromarrays([one, two, three])
np.savetxt('arcgis.csv', rarr, fmt=['%s', '%s', '%f'], delimiter=";")
Improve this answer
Sign up to request clarification or add additional context in comments.

3 Comments

Did you try what I suggested? What does the output of that print statement look like? Once you get that output, post some of it in an edit at the bottom of your question.
Thanks tel! I tried your solution, and the error related to the error message was solved. However, as you said, a new error message appeared.I will post the message in my question
I changed the code. However, there is a new error message related to the "np.rec.fromarrays"
1

The solution that I found was to use the CSV File Reading and Writing library.

I just added the library in the beginning "import csv" and then added the following lines along the code.

First, in the begging, I needed to set the file directory to save the data.

#Opening the file to save the data
write_path = main_directory + "\\Dados\\Exportados\\" + "file_name" + ".csv"
file = open(write_path, 'a')
writer = csv.writer(file)

The 'a' parameter allows you to append to the end of the file

Then, in the step that I wanted the data to be saved, I added the following line:

#Writing to the file
writer.writerow([mes_pelo_nome, ano_pelo_nome, dia_pelo_nome, mascara, media])

In the end, I added the following line to close the file:

file.close()
Improve this answer

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.