Datacollector limitations

TorbenMandrup

Hi There
I have this macro (with a lot of help from Yuri) that makes a few operations on hidden images. Making some counts and logical operations - not very complicated. All data from the counting is stored in the datacollector. My problem is, that I want to process many images (+500) and it seems like the datacollector chokes. When I cross about 5-10K of lines in the datacollector everything goes VERY slow. I have tried to make few lines of code that activates the datacollector, loades data into Excel and empties the datacollector for each image. This helps a litlle bit but after about 500 images the processing it sloooooooooooooow:-(
What can I do to keep my application "clean", because it seems like something is piling up or am I just reaching the limits her?

 - Torben

mbatchelor

Torben --

My approach to resolving this problem would be to make your macro save the information from each SINGLE IMAGE in an ASCII TEXT FILE.  This is "lean and mean" and should not bog down.

After all of your images are processed, you can combine the ASCII TEXT FILES.  This can be done with:

** A little routine that will read each IMAGE DATA FILE and append it to a SUMMARY DATA DATA FILE for the GROUP (FOLDER?)
** OS COMMAND like "COPY *.TXT COMBINED.TEXT"

After you have the SUMMARY DATA FILE, you can do what ever statistics you need to do on the data for the GROUP (FOLDER?) OF IMAGES

This also has a benefit of allowing you to EDIT a SINGLE IMAGE / SINGLE DATA FILE from the GROUP (FOLDER).  If you find a SINGLE IMAGE / SINGLE DATA is a problem:

** Edit (DELETE / MOVE / HIDE) the SINGLE IMAGE / SINGLE DATA from the GROUP (FOLDER)
** Rerun the SUMMARIZE on the SINGLE DATA FILES
** Rerun the STATISTICS on the SUMMARY DATA FILE

I hope this information is helpful.

-- Matt

TorbenMandrup

Okay - That's a nice input. So this is instead of using the datacollector you mean?

 - Torben

mbatchelor

2017-01-06-115354

Torben --

If the DATA COLLECTOR can generate an ASCII DATA FILE under macro control, then your macro can:

** Use the DATA COLLECTOR to generate the SINGLE IMAGE ASCII DATA FILE
** Clear the DATA COLLECTOR

After you have all of the SINGLE IMAGE DATA FILES (from the DATA COLLECTOR)  then:

** Perform the SUMMARY
** Calculate the STATISTICS

I have not used the DATA COLLECTOR in this way.

When I have had similar challenges, I have had my macros:

** Learn the SINGLE IMAGE DATA from the DATA TABLE
** Output SINGLE IMAGE DATA to a SINGLE IMAGE ASCII DATA FILE

then

** Perform the SUMMARY
** Calculate the STATISTICS

I hope this information is helpful.

-- Matt

YuriG

Hi Torben,

About "chocking" of Data Collector: do you collect data of every object? If yes, then you may end up with hundreds of thousands lines in data table, which will definitely  slow down the processing.
You can try collect only stats (using Measurement stats table), you can collect Mean, Max, Min, SdtDev and other stat field of any measurement, so you get only 1 line per image, in that case 500+ images is not a problem.
Another advice is to keep Data Collector table hidden (switched off) until the end of the batch, so you don't spend time for refreshing of data table.

Yuri

TorbenMandrup

I cannot do with just statistics. I end my script with these lines:

        With Measure.Data.Collector.TableCommands.ActivateTab(FD_Measure)

            .TabName = "Measurements"

            .Run()

        End With

        With Measure.Data.Collector.TableCommands.SendToExcel(FD_Measure)

            .UseStatistics = False

            .Run()

        End With

        With Application.RibbonCommands.SelectRibbonTab(FD_Measure)

            .TabName = "Measure"

            .Run()

        End With

        Measure.Data.CollectorCommands.Clear(FD_Measure).Run()

Bu it seems that just using the datacollector is the problem, bescause it doesn't clean the cache(?).

Can I approach the count-data without using datacollector and send them to a txt file or similar?

 - Torben

YuriG

Torben,

I can see that you indeed collect data for every measurement object. What I suggest is to collect only statistics to different table "Measurement Stats".

You will still get the same results, but with much less lines.

What do you mean under " it doesn't clean the cache"? There in no cache, the table just gets too large and refresh takes longer.

You can also save measurement data from every image to a TXT file, but you have to do some aditional programing to define file names and you will end up with 500+ txt files, which is not easy to handle.

I still think that using "Measurement Stats" table of Data Collector is the best solution.

Yuri

TorbenMandrup

I need the actual result for the different particles. I put it into an Excel sheet and perform a complicated macro for further processing. I don't know if it doesn't clear the cache - It's just an expression for cleaning the memory (since it chokes). How can I save results in a txt file without using the datacollector? I would be nice to access the countings in the programme without using the GUI somehow...

 - Torben

YuriG

Torben,

If you need data for every particle, then Stats table cannot be used. Do you keep Data Collector table opened during the batch? If yes, then just close it before running the batch. The data will be still collected correctly, but much faster.
Open data table only after all images are measured.

Yuri

TorbenMandrup

Ahhh interesting - I will try that:-)

TorbenMandrup

Can I send to Excel from the Datacollector 'Measurements' tab without activating (showing/opening) the datacollector? That way I would save some time and prevent loosing results if things are crashing:-)

 - Torben

YuriG

Torben,

Sending data to Excel requires data table, so the command SendToExcel will show the table and then send data to Excel. But if you just want to prevent losing the data, you can save the data to file without showing data table, like this:

    Public Function SaveToFile() As SimpleScript
        SaveToFile = New SimpleScript

        With Measure.Data.CollectorCommands.Save(SaveToFile)
            .FileName = ThisApplication.Path(mcPathType.mcptWriteableDocuments) & "DC1.adoxml"
            .Run()
        End With

    End Function

Then you can load this file to Data Collector's table using the Load button.

Yuri

TorbenMandrup

OMG - What is going on now Yuri. I cannot see my measurements tab in the datacollector!?

Have I gone blind!?

 - Torben

YuriG

Check your measurement list (in Types dropdown). Try to reload it if you have it in a file.

Yuri

TorbenMandrup

I have done so and I get this error:


I have attached my datacollector

YuriG

Hi Torben,

It looks like the relation "Image_Measurements" was missing in your file. It could be removed when you deleted some of the tables. The relation is used in "Display Name" measurement in Measurements table.
I've removed and re-added the measurement. Please check the attached file.

Yuri

mbatchelor

Yuri --

While reading through the comments on this discussion, I noticed the following code:

.FileName = ThisApplication.Path(mcPathType.mcptWriteableDocuments) & "DC1.adoxml"

This raised a flag in the coding part of my brain.

Don't the PATHS generated by PREMIER need a

    \

appended to them before the FILE NAME to work properly?

My experience says that this statement should be

.FileName = ThisApplication.Path(mcPathType.mcptWriteableDocuments) & "\DC1.adoxml"

but I have not tried this out within PREMIER and this PATH may work differently than the ones that I have worked with.

Thanks.

-- Matt

YuriG

Matt, all Path variables have backslash, so you don't need it in the file name.

Yuri

mbatchelor

Yuri --

With a bit of research and a bit more caffeine, I see that you are right.

I am a heavy user of the MACRODIR COMMAND and it does not include its own backslash.

I think this is what caused my confusion.

Thank you for the clarification.

-- Matt