Batch processing - Limitations

TorbenMandrup · November 2016

I have managed to create a macro that makes a few counts, AND operation, count again, serving results in Excel, closing images, bla bla bla. Everything works fine so far. I can even make this work in a batch processing, but now comes the problems.I want to batch process a lot of images (>500) and it seems like the opening/closing of images makes Image-Pro choke. I get a message of exceeded number of GID (???) 9800 and the batch wont go any further.

Have I reach a limit or can I somehow clean up cash as we go along?

- Torben Mandrup

YuriG · November 2016

Hi Torben,

I don't see this problem with StubNo 1 Image 27.png. Can you please attach StubNo 1 Image 245.png, which you use for tests?

Yuri

NikitaRG · November 2016

Hi Torben,

You are reaching Windows limitation of the graphical objects (GDI). Several GDI used for each open image. The main reason of reaching this limit then image is not fully released (some references to the image kept in application or macro).

Could you past your macro?

Thanks,
Nikita.

TorbenMandrup · November 2016

Hi Nikita - Thanks for your reply. You're excactly right. I can see on the batch processing task manager, that closing the last window takes longer and longer and longer as macro is running. This is the macro:

Public Function FD007() As SimpleScript

FD007 = New SimpleScript

Dim doc1, image1, doc2

Dim docList1 = New List(1), doc3, image2, window1, window2, window3

With Application.RibbonCommands.SelectRibbonTab(FD007)

.TabName = "CountSize"

.Run()

End With

With Application.DocumentCommands.Active(FD007)

.Run(doc1)

End With

With Measure.MeasurementsCommands.Options(FD007)

.ActiveClass = 1

.Run(doc1)

End With

With Measure.Measurements.OptionsCommands.Open(FD007)

.FileName = "C:\Users\torbe\Dropbox\Torben Dokumenter\Mandrup Software\Kunder\RW\Ny SEM\Image-Pro\FD_Oprindelig.iqo"

.FilterIndex = 1

.Run(doc1)

End With

With Measure.MeasurementsCommands.ExecuteCount(FD007)

.Run(doc1)

End With

With Measure.Measurements.TableCommands.SendToExcel(FD007)

.UseStatistics = False

.Run(doc1)

End With

With Measure.Measurements.CreateCommands.Mask(FD007)

.Run(doc1, image1)

End With

With Application.DocumentCommands.Activate(FD007)

.Run(image1, doc2)

End With

With Application.DocumentCommands.Open(FD007)

.Filenames = New String() {"C:\Users\torbe\Dropbox\Torben Dokumenter\Mandrup Software\Kunder\RW\Ny SEM\Procesbilleder\FiveLines.bmp"}

.Run(docList1)

End With

With Application.DocumentCommands.Activate(FD007)

.Run(docList1(0), doc3)

End With

With Application.RibbonCommands.SelectRibbonTab(FD007)

.TabName = "Process"

.Run()

End With

With Process.ProcessCommands.Operations(FD007)

.Operation = eOperation.AND

.NumberValue = 0R

.ResultType = eResultType.Apply

.Run(doc3, image1, image2)

End With

With Application.RibbonCommands.SelectRibbonTab(FD007)

.TabName = "CountSize"

.Run()

End With

With Measure.MeasurementsCommands.Options(FD007)

.ActiveClass = 1

.Run(doc3)

End With

With Measure.Measurements.OptionsCommands.Open(FD007)

.FileName = "C:\Users\torbe\Dropbox\Torben Dokumenter\Mandrup Software\Kunder\RW\Ny SEM\Image-Pro\FD_Stumper.iqo"

.FilterIndex = 1

.Run(doc3)

End With

With Measure.MeasurementsCommands.ExecuteCount(FD007)

.Run(doc3)

End With

With Measure.Measurements.TableCommands.SendToExcel(FD007)

.UseStatistics = False

.Run(doc3)

End With

With Measure.MeasurementsCommands.DeleteAll(FD007)

.Run(doc3)

End With

With Application.WindowCommands.Define(FD007)

.Run(doc3, window1)

End With

With Application.WindowCommands.Close(FD007)

.Run(window1)

End With

With Application.DocumentCommands.Activate(FD007)

.Run(doc2, doc2)

End With

With Application.WindowCommands.Define(FD007)

.Run(doc2, window2)

End With

With Application.WindowCommands.Close(FD007)

.Run(window2)

End With

With Application.DocumentCommands.Activate(FD007)

.Run(doc1, doc1)

End With

With Measure.MeasurementsCommands.DeleteAll(FD007)

.Run(doc1)

End With

With Application.WindowCommands.Define(FD007)

.Run(doc1, window3)

End With

With Application.WindowCommands.Close(FD007)

.Run(window3)

End With

End Function

End Module

TorbenMandrup · November 2016

I just ran the macro on 317 images - The first one takes 5 sec and the last one 50 sec. Big problem, but what to do about it?

YuriG · November 2016

Hi Torben,

Your macro looks ok, but to reproduce the problem we will need your option files. Can you please zip all OPT files you use in the macro, FiveLines.bmp and at least one sample image and attach it to your post.

Also, what version of IP Premier do you use? (The latest version is 9.2).

Thanks,

Yuri

TorbenMandrup · November 2016

I am relatively new on this:- ) Yes I am using 9.2 but I have no idea what is OPT files - and where to find them!?
I have attached sample file, FiveLines and iqo files, but I am not sure that's what you're looking for?

- Torben

YuriG · November 2016

Hi Tobren,

Thanks for the provided code and sample images. We were able to reproduce the problem and will work to fix it for the next update.

Meanwhile I can give you some recommendations to make processing faster:
- try to work on invisible images - it will significantly speed up the processing and avoid resource leak.
- in your current macro you export 2 different data tables into Excel for every image, so in the end of batch processing you have a very large Excel sheet with mix of data tables (that large Excel sheet can be the reason of longer processing in the end of the batch). I would recommend to use Data Collector for that purpose, it will give you more structural view of the results (data collector may include several data tables, if necessary). You can also collect only statistics, not data from every segmented objects, if it will work for you.

Best regards,

Yuri

TorbenMandrup · November 2016

Hi Yuri - Thank's for your reply. I am already working with the data collector instead - much better:-))

But when I disable 'Display Documents' in the batchprocessing, nothing happens - the macro is not executed. So what do you mean working on invisible images!?

Regars Torben

YuriG · November 2016

Torben,

Running macro on hidden images requires some modifications to the recorded macro:

1. You add a parameter to the macro that defines the document loaded by batch processing, like this:

    Public Function LoopOnHidden(doc1 As Object) As SimpleScript

2. be sure to remove doc1 from the list of internal variables, so doc1 is the document ID that will be processed by the macro.

3. If you have functions inside the macro that load or create new images, add .Visible=false property, so the new image will be loaded or created without showing it:

        With Measure.Measurements.CreateCommands.Mask(LoopOnHidden)
            'create invisible image
            .Visible=False
            .Run(doc1, image1)
        End With

        With Application.DocumentCommands.Open(LoopOnHidden)
            .Filenames = New String() {ThisApplication.Path(mcPathType.mcptConfigurationFiles) & "FiveLines.bmp"}
            'open invisible image
            .Visible=False
            .Run(docList1)
        End With

4. Collect data to Data Collector instead of saving data tables (data tables handle only visible images). Use doc ID to collect the data, like this:

        'collect data
        With Measure.Data.CollectorCommands.Collect(LoopOnHidden)
            .Run(New List({doc1}))
        End With

5. Close all images in the end of the macro using CodeCommand:

        With Automate.ScriptingCommands.CodeCommand(LoopOnHidden)
            If .Run() Then
                ' User Code Here
                ' close all images, avoid user prompt clearing Modified flag
                doc3.Data.Modified=False
                doc3.Data.Close
                doc1.Data.Modified=False
                doc1.Data.Close
                image1.Modified=False
                image1.Close
                ThisApplication.Documents.CloseAll
            End If
        End With

I've attached the complete project. Note, that you will have to setup proper measurements to collect for Data Collector, you can either do it manually before running the batch or in a macro that is set as "Bun Before" in batch dialog.

Run the batch with "Display Documents" off.

Regards,

Yuri

TorbenMandrup · November 2016

Thanks a lot - I will try this tomorrow:-)))

TorbenMandrup · November 2016

It works - in theory - and is faster by the way:-). But this is VERY strange, because it gives me different results in the datacollector depending on the visibility or not. How the heck is that possible!?

I have attached a comparison of the two measurements. But the strangest thing is, that if run your macro and set visibility=True I get the same results (the right ones) as with the original macro.

- Torben

YuriG · November 2016

Hi Torben,

I checked it and found that loading IQO options from file doesn't set the threshold if the image is not visible, so I've added a line that sets threshold explicitly after loading IQO (the same threshold as in IQO file):

        With Measure.Measurements.OptionsCommands.Open(LoopOnHidden)
            .FileName = ThisApplication.Path(mcPathType.mcptConfigurationFiles) & "FD_Oprindelig.iqo"
            .FilterIndex = 1
            .Run(doc1)
        End With

        With Measure.ThresholdToolCommands.Thresholds(LoopOnHidden)
            .AllowOverlap = False
            .Interpretation = eInterpretation.Mono
            .Classes = New System.Collections.Generic.List(Of SegmentationClass)
            .Classes.Add(New SegmentationClass("Class 1",System.Drawing.Color.Blue,New Double(){139R,255R}))
            .Run(doc1)
        End With

I've attached a new version of the project.

Regards,

Yuri

TorbenMandrup · November 2016

You're a wizard:-) Now I get similar results for the original image but for processed Fivelines_AND_Mask image there still the difference:-(

- Torben

Image: https://us.v-cdn.net/5020878/uploads/editor/p2/6pbt2y7498k8.png

TorbenMandrup · November 2016

Image: https://us.v-cdn.net/5020878/uploads/editor/ku/mg5pftfhtm3b.png

TorbenMandrup · November 2016

....but it's great - it's REALLY fast...three times fast appr

TorbenMandrup · November 2016

All the "Line:End Y" must be 0,5,10,15,20 be definition. Because they are AND'ed from the five lines.

- Torben

YuriG · November 2016

Hi Tobren,

I tried your image 245, but I get correct results:

Image: https://us.v-cdn.net/5020878/uploads/editor/bx/ff1fahq3lwuj.jpg

Please try to run the batch from my project (without any editing), making LoopOnHidden as the batch macro and processing a folder with one image 245 to check if you have the same results.

Yuri

TorbenMandrup · November 2016

That is pretty weird - It works. I get slightly different result in the end, but I will look into that. That may be my processing of numbers...
- Torben

TorbenMandrup · November 2016

Hmmm, there is obviously something I don't undertand about projects here. Does your module1.vb exists as a seperate file (I cannot find it) or is it somehow embedded into the project file?
- Torben

YuriG · November 2016

Torben,

IPX file contains all project files including Module1.vb (it's a ZIP). That's the best way redistributing Premier macro projects.

Yuri

TorbenMandrup · November 2016

Nice - But how can I unwrap it - If I want to change for instance the configuration files (threshold etc) - Is there a best practice tutorial some where?

YuriG · November 2016

Torben,

You can unwrap the project file in the Project Workbench using File | Save | Unpack Project Package.
You can find more info in Project Workbench help (accessible from the help button dropdown of the Project workbench).

There are also several tutorials on this page http://www.mediacy.com/imagepropremier/learn

Yuri

TorbenMandrup · November 2016

This is great - Everything seems to work fine. I would like to use the option of monitoring a folder (I have a SEM microscope feeding images into). Is there something in this macro that disables that option?
- Torben

YuriG · November 2016

Hi Torben,

Monitor Folders option is available for any macro, it will process images as soon as they appear in the folder.

Yuri

TorbenMandrup · November 2016

Hmmm doesn't seem to work right - Is there an settings I should be aware of?

YuriG · November 2016

No, there is no additional options. Select the folder, activate the Monitor Folders checkbox and click the Start button. Then copy a file to your folder - it should be processed by the LoopOn macro.
I can confirm that LoopOnHidden works with that option.

Yuri

TorbenMandrup · January 2017

Sorry - To bother you again Yuri. I can manage to unwrap project files into another folder. But how do I get it back. I simply can't make it work, it's like your code is hidden in a box I can't get into!?

I have tried to make a new module, but it gives a different result than yours (LoopOnHidden) - I cannot figure out why.

- Torben

TorbenMandrup · January 2017

I may be a similar thing as for the threshold setting. I am trying to change the ranges when I do the counting, perhaps that is not loaded either when working on hidden images?

- Torben

YuriG · January 2017

Hi Torben,

You can "unwrap" IPX project using the Unpack Project package menu item in the Project workbench menu:

Image: https://us.v-cdn.net/5020878/uploads/editor/ox/92nu17u141ff.jpg

Threshold tool recording also works with visible images, so you have to modify the macro to make it work on invisible.

Yuri

mbatchelor · August 2017

2017-08-02-152044

Yuri --

If an IMAGE is OPEN and VISIBLE, can it be made INVISIBLE?

If YES, can you please provide a ROUTINE to do this?

My attempt is below but it seems to CLOSE the VISIBLE IMAGE rather than making it INVISIBLE.

If NO, I guess the OPEN IMAGE must be SAVED, CLOSED, and then OPENED (as INVISIBLE).

Thanks.

-- Matt

*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-

    Public Function Make_Image_Invisible2()

        Debug.Print "ThisApplication.Documents.Values.Count"
        Debug.Print ThisApplication.Documents.Values.Count

        ThisApplication.ActiveImage.Visible = False

        Debug.Print "ThisApplication.Documents.Values.Count"
        Debug.Print ThisApplication.Documents.Values.Count

    End Function

Batch processing - Limitations

Best Answer

Answers