Batch processing slows down

fsup

An image is opened invisibly and then closed. With "Debugging with UI V1A" you can see that the image remains open (All Images Count = 1).

    Public Sub OpenStandardImage

        Dim imageDoc As IMcDocument = Nothing
        Dim image As McImage = Nothing
        Dim docList As System.Collections.Generic.List(Of IMcDocument)

        With MediaCy.Automation.Application.DocumentCommands.Open(Nothing)
            .Filenames = New String() {"C:\Users\Public\Documents\Image-Pro Demo Images\Color\Tumor.tif"}
            .Visible = False
            .Run(docList)
        End With

        imageDoc = docList(0)
        image = TryCast(imageDoc.Data,McImage)

        docList = Nothing

        image.Modified = False
        image.Close
        image = Nothing

        imageDoc = Nothing

    End Sub

The same thing happens with czi's (ImageSet). By the way - how can I close invisible ImageSets? Because there exists no close method as in McImages.

Image Pro Version: 10.0.4 Build 6912

Thank you in advance.

fsup

YuriG

Hi fsup,

Thank you for providing the macro and images. The macro looks good and if you run GarbageCollect (from Debugging project) after executing of StartImageProcessing macro all images are released properly and memory restored.
You use several macro commands that are executed inside your macros and the problem is that the commands are not disposed and hold reference to image as input parameter until the macro StartImageProcessing exits. 
There are several ways to avoid holding references in command inputs, such as using Execute instead of Run, or calling.Dispose after .Run, but I can suggest calling ThisApplication.ProcessAll to clear all commands and then running garbage collection.
Here is the code (in bold) that you should add in your loop macro to clear all resources properly.

...
            _imageExt = Nothing

            _imageSet = Nothing

            ThisApplication.Documents.Remove(_imageDoc.Data)
            _imageDoc.OnClose()
            _imageDoc = Nothing

            'cleanup all commands
            ThisApplication.ProcessAll
            'garbage collect
            System.GC.Collect()
            System.GC.WaitForPendingFinalizers()
            System.GC.Collect()
            System.GC.WaitForPendingFinalizers()

            Dim CollectionImagesCount As Integer, AllImagesCount As Integer
            CountImages2(CollectionImagesCount, AllImagesCount)

            endTime = System.DateTime.Now
...

The result output looks like this:

Process started: 8/29/2019 4:47:44 PMLoop: 1, CollectionImagesCount: 0, AllImagesCount: 0, Runtime(s): 68.815Loop: 2, CollectionImagesCount: 0, AllImagesCount: 0, Runtime(s): 64.683Process finished: 8/29/2019 4:49:58 PM

Let me know if it fixes the problem.

Regards,

Yuri

YuriG

Hi fsup,

It looks like the macro ran for 6 hours processing 200 of 3GB images. Image-Pro has some bookkeeping lists, such as Audit Trail, that starts from the beginning of the session and then constantly grows until the program is closed. It doesn't have any significant memory footprint and shouldn't affect processing speed, but you can try switching it off ("Enable Audit Trail" option on the Advanced tab of the application options) and see if it helps. 

Yuri

YuriG

Hi fsup,

If you open the image as a document, you have to remove is as a document, like this:

    Public Sub OpenStandardImage

        Dim imageDoc As IMcDocument = Nothing
        Dim image As McImage = Nothing
        Dim docList As System.Collections.Generic.List(Of IMcDocument)

        With MediaCy.Automation.Application.DocumentCommands.Open(Nothing)
            .Filenames = New String() {"C:\Users\Public\Documents\Image-Pro Demo Images\Color\Tumor.tif"}
            .Visible = False
            .Run(docList)
        End With

        imageDoc = docList(0)
        ThisApplication.Documents.Remove(imageDoc.Data)
        imageDoc.OnClose()
    End Sub

If you just want to have an image, you can open it directly using Images.Open:

Public Sub OpenStandardImage2
        Dim image As McImage = ThisApplication.Images.Open("C:\Users\Public\Documents\Image-Pro Demo Images\Color\Tumor.tif",mcImageCreateFlags.mcicfNotVisible Or mcImageCreateFlags.mcicfNoAddToCollection)
        'do something with image

        'close it
        image.Close
    End Sub

Yuri

fsup

Hi Yuri,

I did some tests with czi images. The batch loads a czi image (~3GB), extract a channel and do some measurements on the extracted channel. If I increase the number of loops (set variable noOfLoops to 50) the batch slows down and an error "insufficient memory" appears in ImagePro (depending on the size of ram) after a while. In the Windows Task Manager you can see that the ram increases. To analyse the problem I integrated CountImages2 and System.Windows.Forms.Application.DoEvents after each loop.

AllImagesCount in CountImages2 shows that an image in each loop stays opened. I think thats the problem why the batch slows down.

How can I fix this? Do you have an idea?

Attached you can find the project which I use for testing. Where can I upload the big sized czi image?

Thanks in advance.

fsup

YuriG

Hi fsup,

You can upload the image here: 

Click here to upload files.

Yuri

fsup

Hi Yuri, 

I have uploaded an zipped example image.

Thanks for your help

fsup

fsup

Hi Yuri,

I tested your solution. All images are closed. I repeated the test over night with a higher number of passes (noOfLoops = 200). But the process slows down:

Process started: 02.09.2019 07:27:41
Loop: 1, CollectionImagesCount: 0, AllImagesCount: 0, Runtime(s): 77,978
..
Loop: 199, CollectionImagesCount: 0, AllImagesCount: 0, Runtime(s): 120,821
Loop: 200, CollectionImagesCount: 0, AllImagesCount: 0, Runtime(s): 119,93
Process finished: 02.09.2019 13:20:38

What could be the cause? How can I fix this?

Thank you in advance

fsup

fsup

Hi Yuri,

thank you for supporting me. First tests looking very good.

fsup