[Gwyddion-users] NaN value in statistical measurment

Discussion:

Moriceau, Julien

2015-01-06 15:35:23 UTC

Hello,
It is my first message to you guys because I am struggling with gwyddion (I precise I am a new user).
I have AFM files that I want to analyze, but a lot of these files are containing NaN value.
I can open these images without too much trouble but the software cannot calculate any of the advanced statistical data (as HHCF that I am interested in) because, I guess, of the NaN values.
Is there any way to correct this problem? Or not to take in account NaN during calculation? (the calculation is working well with the height distribution for example).
Thanks a lot for your help.
Regards,
--
Julien

David Nečas (Yeti)

2015-01-06 19:15:40 UTC

Permalink

Post by Moriceau, Julien
It is my first message to you guys because I am struggling with
gwyddion (I precise I am a new user).
I have AFM files that I want to analyze, but a lot of these files are containing NaN value.
I can open these images without too much trouble but the software
cannot calculate any of the advanced statistical data (as HHCF that I
am interested in) because, I guess, of the NaN values.
Is there any way to correct this problem? Or not to take in account
NaN during calculation? (the calculation is working well with the
height distribution for example).

Moriceau, Julien

2015-01-07 21:13:12 UTC

Permalink

Thank you a lot for your quick response.
I applied your methods which worked well. I was importing a .txt file with 3 rows (XYZ).
I have one more question.
I was wondering if it was possible to save a file with the same organization as my original file, meaning XYZ (in case of I want to reopen it later with Excel for example), because .gwy is not really convenient.
Because after modification I tried to save it (with the "save as" command) in .txt but what I obtained was weird. I guess because I saved pixels of the picture and not the original value XYZ.
I don't know if I am clear, please tell me if not.
Thank you for your help.
Regards
--
Julien

PS: your software is awesome!

-----Original Message-----
From: David Nečas (Yeti) [mailto:***@gwyddion.net]
Sent: Tuesday, January 06, 2015 2:16 PM
To: Gwyddion use discussion
Subject: Re: [Gwyddion-users] NaN value in statistical measurment

Hello,

Gwyddion does not work with NaNs. Some function may do something meaningful when encounterng data containing NaNs, but only by coincidence, not by design.

The normal handling is to replace undefined values with some ‘neutral’
value upon import and create a mask covering the invalid pixels. It is possible that this treatment is missing in the import of some file format where we did not expect NaNs to occur. Please post in which file format you encounter the NaNs so that we can fix its import.

When you have the data with mask over invalid pixels you can
- exclude the masked area from processing in functions that support this
- apply some interpolation or arithmetic function to replace the masked
pixels with other suitable values
- use the entire image in functions that require a regular area (most of
the advanced ones)

Regards,

Yeti

------------------------------------------------------------------------------
Dive into the World of Parallel Programming! The Go Parallel Website, sponsored by Intel and developed in partnership with Slashdot Media, is your hub for all things parallel software development, from weekly thought leadership blogs to news, videos, case studies, tutorials and more. Take a look and join the conversation now. http://goparallel.sourceforge.net _______________________________________________
Gwyddion-users mailing list
Gwyddion-***@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/gwyddion-users

David Nečas (Yeti)

2015-01-08 07:54:08 UTC

Permalink

Post by Moriceau, Julien
I applied your methods which worked well. I was importing a .txt file with 3 rows (XYZ).

OK, for a vendor file file format we could improve the undefined value
handling but for raw text you are on your own...

Post by Moriceau, Julien
I was wondering if it was possible to save a file with the same
organization as my original file, meaning XYZ (in case of I want to
reopen it later with Excel for example), because .gwy is not really
convenient.

This should not be a big problem to add. Gwyddion can export (apart
from VTK which purely is for visualisation) data to this xyz format

http://gwyddion.net/documentation/user-guide-en/gxyzf.html

It is still binary though, so no luck loading it to MS Excel.

Post by Moriceau, Julien
Because after modification I tried to save it (with the "save as"
command) in .txt but what I obtained was weird. I guess because I
saved pixels of the picture and not the original value XYZ.

Well, yes, it is saved as a value matrix – this is the normal
representation of raster data (used in all software I know).
Representing data on a regular grid as XYZ throws away the implicit
geometrical information and multiplies the data size by 3 for no good
reason. On the other hand, if you really have a general point cloud
then Gwyddion may not be the right tool since it works with raster data.

Regards,

Yeti

David Nečas (Yeti)

2015-01-08 14:21:06 UTC

Permalink

Post by David NeÄas (Yeti)

Post by Moriceau, Julien
I was wondering if it was possible to save a file with the same
organization as my original file, meaning XYZ...

This should not be a big problem to add.

Added. It will appear in tomorrow's or later development snapshot.
Select ‘XYZ text data’ as the export file format, or just use the .xyz
extension to choose it.

Regards,

Yeti

Moriceau, Julien

2015-01-08 14:23:13 UTC

Permalink

That's great.
Thank you so much!
Regards
--
Julien

-----Original Message-----
From: David Nečas (Yeti) [mailto:***@gwyddion.net]
Sent: Thursday, January 08, 2015 9:21 AM
To: Gwyddion use discussion
Subject: Re: [Gwyddion-users] NaN value in statistical measurment

Post by David NeÄas (Yeti)

Post by Moriceau, Julien
I was wondering if it was possible to save a file with the same
organization as my original file, meaning XYZ...

This should not be a big problem to add.

Added. It will appear in tomorrow's or later development snapshot.
Select ‘XYZ text data’ as the export file format, or just use the .xyz extension to choose it.

Regards,

Yeti

------------------------------------------------------------------------------
Dive into the World of Parallel Programming! The Go Parallel Website, sponsored by Intel and developed in partnership with Slashdot Media, is your hub for all things parallel software development, from weekly thought leadership blogs to news, videos, case studies, tutorials and more. Take a look and join the conversation now. http://goparallel.sourceforge.net _______________________________________________
Gwyddion-users mailing list
Gwyddion-***@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/gwyddion-users

Stefania Carapezzi

2015-01-09 16:04:16 UTC

Permalink

Hi. I'm working with AFM images as well. I was in need to get my image as a
.txt file with 3 rows (XYZ) for further analysis. I did this by saving from
gwiddyon as .txt and then using the matrix file as input in a python
program that output the .txt file with XYZ rows. the same of course can be
done in matlab, et similia. if you wish I can send you my python script, or
I can briefly describe the algorythm (very short).
best regards
Stefania Carapezzi

On Thu, Jan 8, 2015 at 3:23 PM, Moriceau, Julien <

Post by Moriceau, Julien
That's great.
Thank you so much!
Regards
--
Julien
-----Original Message-----
Sent: Thursday, January 08, 2015 9:21 AM
To: Gwyddion use discussion
Subject: Re: [Gwyddion-users] NaN value in statistical measurment

Post by David NeÄas (Yeti)

Post by Moriceau, Julien
I was wondering if it was possible to save a file with the same
organization as my original file, meaning XYZ...

This should not be a big problem to add.

Moriceau, Julien

2015-01-13 14:46:52 UTC

Permalink

Hello,
Thank you for your spontaneous help.
I would be interested to know how you did this python script. I would be really glad if you can explain it here.
Thank you for your help.
Regards
--
Julien

From: Stefania Carapezzi [mailto:***@gmail.com]
Sent: Friday, January 09, 2015 11:04 AM
To: Gwyddion use discussion
Subject: Re: [Gwyddion-users] NaN value in statistical measurment

Hi. I'm working with AFM images as well. I was in need to get my image as a .txt file with 3 rows (XYZ) for further analysis. I did this by saving from gwiddyon as .txt and then using the matrix file as input in a python program that output the .txt file with XYZ rows. the same of course can be done in matlab, et similia. if you wish I can send you my python script, or I can briefly describe the algorythm (very short).
best regards
Stefania Carapezzi

On Thu, Jan 8, 2015 at 3:23 PM, Moriceau, Julien <***@saint-gobain.com<mailto:***@saint-gobain.com>> wrote:
That's great.
Thank you so much!
Regards
--
Julien

-----Original Message-----
From: David NeÄas (Yeti) [mailto:***@gwyddion.net<mailto:***@gwyddion.net>]
Sent: Thursday, January 08, 2015 9:21 AM
To: Gwyddion use discussion
Subject: Re: [Gwyddion-users] NaN value in statistical measurment

Post by David NeÄas (Yeti)

Post by Moriceau, Julien
I was wondering if it was possible to save a file with the same
organization as my original file, meaning XYZ...

This should not be a big problem to add.

Added. It will appear in tomorrow's or later development snapshot.
Select âXYZ text dataâ as the export file format, or just use the .xyz extension to choose it.

Regards,

Yeti

------------------------------------------------------------------------------
Dive into the World of Parallel Programming! The Go Parallel Website, sponsored by Intel and developed in partnership with Slashdot Media, is your hub for all things parallel software development, from weekly thought leadership blogs to news, videos, case studies, tutorials and more. Take a look and join the conversation now. http://goparallel.sourceforge.net _______________________________________________
Gwyddion-users mailing list
Gwyddion-***@lists.sourceforge.net<mailto:Gwyddion-***@lists.sourceforge.net>
https://lists.sourceforge.net/lists/listinfo/gwyddion-users
------------------------------------------------------------------------------
Dive into the World of Parallel Programming! The Go Parallel Website,
sponsored by Intel and developed in partnership with Slashdot Media, is your
hub for all things parallel software development, from weekly thought
leadership blogs to news, videos, case studies, tutorials and more. Take a
look and join the conversation now. http://goparallel.sourceforge.net
_______________________________________________
Gwyddion-users mailing list
Gwyddion-***@lists.sourceforge.net<mailto:Gwyddion-***@lists.sourceforge.net>
https://lists.sourceforge.net/lists/listinfo/gwyddion-users

Stefania Carapezzi

2015-01-14 08:49:50 UTC

Permalink

Hi.
Let's say that you save an AFM image of size s and resolution r in txt
format. Then your txt file contains a r*r matrix. The (i,j) element of the
matrix is the z-value located at x = (i - 1) * step, y = (j - 1) * step,
for step = s / r, for i,j = 1, ...r. This following the gwyddion rule which
sets the (1,1) element of your matrix as the origin of plane x,y.

This is the python script:
res = 512. # number of pixels (float kind)
size = 5000. # size of (squared) image in nm (float kind)
length_step = size / res
f1 = open('H.txt') # input file: topographic image by Gwyddion
f2 = open('H-XYZ.txt', 'w') # output file
linesA = f1.readlines() # read all the rows of the input image
x_ind = 0 # index of rows
for lineA in linesA:
lineA = lineA.replace('\n', '') # remove end line character from row
line
z_values = lineA.split('\t') # split the row line in the z-values
x_data = str(x_ind * length_step) # x coordinate in nm
x_ind += 1
y_ind = 0 # index of columns
for z_value in z_values:
y_data = str(y_ind * length_step) # y coordinate in nm
data = x_data + '\t' + y_data + '\t' + str(float(z_value) * 1E9) +
'\n' # x, y, z coordinates in nm, correspondent to the element (x_ind + 1,
y_ind + 1) of the original matrix
f2.write(data)
y_ind += 1
f1.close()
f2.close()

Best regards
Stefania

On Tue, Jan 13, 2015 at 3:46 PM, Moriceau, Julien <

Post by Moriceau, Julien
Hello,
Thank you for your spontaneous help.
I would be interested to know how you did this python script. I would be
really glad if you can explain it here.
Thank you for your help.
Regards
--
Julien
*Sent:* Friday, January 09, 2015 11:04 AM
*To:* Gwyddion use discussion
*Subject:* Re: [Gwyddion-users] NaN value in statistical measurment
Hi. I'm working with AFM images as well. I was in need to get my image
as a .txt file with 3 rows (XYZ) for further analysis. I did this by saving
from gwiddyon as .txt and then using the matrix file as input in a python
program that output the .txt file with XYZ rows. the same of course can be
done in matlab, et similia. if you wish I can send you my python script, or
I can briefly describe the algorythm (very short).
best regards
Stefania Carapezzi
On Thu, Jan 8, 2015 at 3:23 PM, Moriceau, Julien <
That's great.
Thank you so much!
Regards
--
Julien
-----Original Message-----
Sent: Thursday, January 08, 2015 9:21 AM
To: Gwyddion use discussion
Subject: Re: [Gwyddion-users] NaN value in statistical measurment

Post by David NeÄas (Yeti)

Post by Moriceau, Julien
I was wondering if it was possible to save a file with the same
organization as my original file, meaning XYZ...

This should not be a big problem to add.

Added. It will appear in tomorrow's or later development snapshot.
Select âXYZ text dataâ as the export file format, or just use the .xyz
extension to choose it.
Regards,
Yeti
------------------------------------------------------------------------------
Dive into the World of Parallel Programming! The Go Parallel Website,
sponsored by Intel and developed in partnership with Slashdot Media, is
your hub for all things parallel software development, from weekly thought
leadership blogs to news, videos, case studies, tutorials and more. Take a
look and join the conversation now. http://goparallel.sourceforge.net
_______________________________________________
Gwyddion-users mailing list
https://lists.sourceforge.net/lists/listinfo/gwyddion-users
------------------------------------------------------------------------------
Dive into the World of Parallel Programming! The Go Parallel Website,
sponsored by Intel and developed in partnership with Slashdot Media, is your
hub for all things parallel software development, from weekly thought
leadership blogs to news, videos, case studies, tutorials and more. Take a
look and join the conversation now. http://goparallel.sourceforge.net
_______________________________________________
Gwyddion-users mailing list
https://lists.sourceforge.net/lists/listinfo/gwyddion-users
------------------------------------------------------------------------------
New Year. New Location. New Benefits. New Data Center in Ashburn, VA.
GigeNET is offering a free month of service with a new server in Ashburn.
Choose from 2 high performing configs, both with 100TB of bandwidth.
Higher redundancy.Lower latency.Increased capacity.Completely compliant.
http://p.sf.net/sfu/gigenet
_______________________________________________
Gwyddion-users mailing list
https://lists.sourceforge.net/lists/listinfo/gwyddion-users

Moriceau, Julien

2015-01-14 13:18:43 UTC

Permalink

Hi,
Thank you very much, it will help me a lot.
Regards
--
Julien

From: Stefania Carapezzi [mailto:***@gmail.com]
Sent: Wednesday, January 14, 2015 3:50 AM
To: Gwyddion use discussion
Subject: Re: [Gwyddion-users] NaN value in statistical measurment

Hi.
Let's say that you save an AFM image of size s and resolution r in txt format. Then your txt file contains a r*r matrix. The (i,j) element of the matrix is the z-value located at x = (i - 1) * step, y = (j - 1) * step, for step = s / r, for i,j = 1, ...r. This following the gwyddion rule which sets the (1,1) element of your matrix as the origin of plane x,y.

This is the python script:
res = 512. # number of pixels (float kind)
size = 5000. # size of (squared) image in nm (float kind)
length_step = size / res
f1 = open('H.txt') # input file: topographic image by Gwyddion
f2 = open('H-XYZ.txt', 'w') # output file
linesA = f1.readlines() # read all the rows of the input image
x_ind = 0 # index of rows
for lineA in linesA:
lineA = lineA.replace('\n', '') # remove end line character from row line
z_values = lineA.split('\t') # split the row line in the z-values
x_data = str(x_ind * length_step) # x coordinate in nm
x_ind += 1
y_ind = 0 # index of columns
for z_value in z_values:
y_data = str(y_ind * length_step) # y coordinate in nm
data = x_data + '\t' + y_data + '\t' + str(float(z_value) * 1E9) + '\n' # x, y, z coordinates in nm, correspondent to the element (x_ind + 1, y_ind + 1) of the original matrix
f2.write(data)
y_ind += 1
f1.close()
f2.close()

Best regards
Stefania

On Tue, Jan 13, 2015 at 3:46 PM, Moriceau, Julien <***@saint-gobain.com<mailto:***@saint-gobain.com>> wrote:
Hello,
Thank you for your spontaneous help.
I would be interested to know how you did this python script. I would be really glad if you can explain it here.
Thank you for your help.
Regards
--
Julien

From: Stefania Carapezzi [mailto:***@gmail.com<mailto:***@gmail.com>]
Sent: Friday, January 09, 2015 11:04 AM

To: Gwyddion use discussion
Subject: Re: [Gwyddion-users] NaN value in statistical measurment

Hi. I'm working with AFM images as well. I was in need to get my image as a .txt file with 3 rows (XYZ) for further analysis. I did this by saving from gwiddyon as .txt and then using the matrix file as input in a python program that output the .txt file with XYZ rows. the same of course can be done in matlab, et similia. if you wish I can send you my python script, or I can briefly describe the algorythm (very short).
best regards
Stefania Carapezzi

On Thu, Jan 8, 2015 at 3:23 PM, Moriceau, Julien <***@saint-gobain.com<mailto:***@saint-gobain.com>> wrote:
That's great.
Thank you so much!
Regards
--
Julien

-----Original Message-----
From: David NeÄas (Yeti) [mailto:***@gwyddion.net<mailto:***@gwyddion.net>]
Sent: Thursday, January 08, 2015 9:21 AM
To: Gwyddion use discussion
Subject: Re: [Gwyddion-users] NaN value in statistical measurment

Post by David NeÄas (Yeti)

Post by Moriceau, Julien
I was wondering if it was possible to save a file with the same
organization as my original file, meaning XYZ...

This should not be a big problem to add.

Mark S. Bentley

2015-01-16 14:38:13 UTC

Permalink

Dear all,

I currently generate Gwyddion data files from an instrument and use the
per channel mask to flag "bad" pixels. I now also want to add a second
indicator that flags a different (boolean) property per pixel (another
type of good/bad, but distinct from the first).

I assume that only one mask per channel is allowed? Is there a sensible
way to flag another set of boolean per-pixel properties for a given data
channel? From the docs, boolean arrays are not permitted. I could use
another array type, but (a) my file sizes increase further (another
array of 32-bit elements) and (b) this would not be a property of the
data channel per say.

Any suggestions welcome!

Regards,

Mark

David Nečas (Yeti)

2015-01-17 22:13:55 UTC

Permalink

Post by Mark S. Bentley
I currently generate Gwyddion data files from an instrument and use the
per channel mask to flag "bad" pixels. I now also want to add a second
indicator that flags a different (boolean) property per pixel (another
type of good/bad, but distinct from the first).
I assume that only one mask per channel is allowed? Is there a sensible
way to flag another set of boolean per-pixel properties for a given data
channel? From the docs, boolean arrays are not permitted. I could use
another array type, but (a) my file sizes increase further (another
array of 32-bit elements) and (b) this would not be a property of the
data channel per say.

Are you talking about the user interface or internal representation?

Anyway, there is indeed one mask per channel. Furthermore, mask data
are represented as doubles even though nowadays all functions use them
as binary values. This is for historical reasons and can't be really
changed without breaking GWY files seriously (the idea of mask values
between 0 and 1, which is useful for instance in the GIMP, did not turn
out to be so useful in AFM data processing). So, when file sizes are
considered, it sucks. Fortunately, the Moore's law for storage size has
a shorter time scale that the corresponding law for CPU power so it is
not that much a concern...

There are some tricks to use multiple masks. ‘Extract mask’ and ‘Mark
with’ are your best friends. The first creates a channel containing the
mask. The latter can set/modify the mask of a channel using another
channel/mask and some logical operators. It may be less convenient than
if masks were first-class entities with their own identity, but
incidentally I did rather a lot of these things recently and it works
quite well.

Regards,

Yeti