Batch processing slows down

An image is opened invisibly and then closed. With "Debugging with UI V1A" you can see that the image remains open (All Images Count = 1).


    Public Sub OpenStandardImage

        Dim imageDoc As IMcDocument = Nothing
        Dim image As McImage = Nothing
        Dim docList As System.Collections.Generic.List(Of IMcDocument)

        With MediaCy.Automation.Application.DocumentCommands.Open(Nothing)
            .Filenames = New String() {"C:\Users\Public\Documents\Image-Pro Demo Images\Color\Tumor.tif"}
            .Visible = False
            .Run(docList)
        End With

        imageDoc = docList(0)
        image = TryCast(imageDoc.Data,McImage)

        docList = Nothing

        image.Modified = False
        image.Close
        image = Nothing

        imageDoc = Nothing

    End Sub

The same thing happens with czi's (ImageSet). By the way - how can I close invisible ImageSets? Because there exists no close method as in McImages.

Image Pro Version: 10.0.4 Build 6912

Thank you in advance.
fsup


Best Answers

  • Accepted Answer
    Hi fsup,

    Thank you for providing the macro and images. The macro looks good and if you run GarbageCollect (from Debugging project) after executing of StartImageProcessing macro all images are released properly and memory restored.
    You use several macro commands that are executed inside your macros and the problem is that the commands are not disposed and hold reference to image as input parameter until the macro StartImageProcessing exits. 
    There are several ways to avoid holding references in command inputs, such as using Execute instead of Run, or calling.Dispose after .Run, but I can suggest calling ThisApplication.ProcessAll to clear all commands and then running garbage collection.
    Here is the code (in bold) that you should add in your loop macro to clear all resources properly.

    ...
                _imageExt = Nothing
    
                _imageSet = Nothing
    
                ThisApplication.Documents.Remove(_imageDoc.Data)
                _imageDoc.OnClose()
                _imageDoc = Nothing
    
                'cleanup all commands
                ThisApplication.ProcessAll
                'garbage collect
                System.GC.Collect()
                System.GC.WaitForPendingFinalizers()
                System.GC.Collect()
                System.GC.WaitForPendingFinalizers()
    
                Dim CollectionImagesCount As Integer, AllImagesCount As Integer
                CountImages2(CollectionImagesCount, AllImagesCount)
    
                endTime = System.DateTime.Now
    ...

    The result output looks like this:

    Process started: 8/29/2019 4:47:44 PM
    Loop: 1, CollectionImagesCount: 0, AllImagesCount: 0, Runtime(s): 68.815
    Loop: 2, CollectionImagesCount: 0, AllImagesCount: 0, Runtime(s): 64.683
    Process finished: 8/29/2019 4:49:58 PM


    Let me know if it fixes the problem.

    Regards,

    Yuri
  • Accepted Answer
    Hi fsup,

    It looks like the macro ran for 6 hours processing 200 of 3GB images. Image-Pro has some bookkeeping lists, such as Audit Trail, that starts from the beginning of the session and then constantly grows until the program is closed. It doesn't have any significant memory footprint and shouldn't affect processing speed, but you can try switching it off ("Enable Audit Trail" option on the Advanced tab of the application options) and see if it helps. 

    Yuri

Answers

  • Hi fsup,

    If you open the image as a document, you have to remove is as a document, like this:

        Public Sub OpenStandardImage
    
            Dim imageDoc As IMcDocument = Nothing
            Dim image As McImage = Nothing
            Dim docList As System.Collections.Generic.List(Of IMcDocument)
    
            With MediaCy.Automation.Application.DocumentCommands.Open(Nothing)
                .Filenames = New String() {"C:\Users\Public\Documents\Image-Pro Demo Images\Color\Tumor.tif"}
                .Visible = False
                .Run(docList)
            End With
    
            imageDoc = docList(0)
            ThisApplication.Documents.Remove(imageDoc.Data)
            imageDoc.OnClose()
        End Sub
    

    If you just want to have an image, you can open it directly using Images.Open:

    Public Sub OpenStandardImage2
            Dim image As McImage = ThisApplication.Images.Open("C:\Users\Public\Documents\Image-Pro Demo Images\Color\Tumor.tif",mcImageCreateFlags.mcicfNotVisible Or mcImageCreateFlags.mcicfNoAddToCollection)
            'do something with image
    
            'close it
            image.Close
        End Sub
    

    Yuri
  • Hi Yuri,

    I did some tests with czi images. The batch loads a czi image (~3GB), extract a channel and do some measurements on the extracted channel. If I increase the number of loops (set variable noOfLoops to 50) the batch slows down and an error "insufficient memory" appears in ImagePro (depending on the size of ram) after a while. In the Windows Task Manager you can see that the ram increases. To analyse the problem I integrated CountImages2 and System.Windows.Forms.Application.DoEvents after each loop.

    AllImagesCount in CountImages2 shows that an image in each loop stays opened. I think thats the problem why the batch slows down.



    How can I fix this? Do you have an idea?

    Attached you can find the project which I use for testing. Where can I upload the big sized czi image?

    Thanks in advance.
    fsup

  • Hi fsup,

    You can upload the image here: 

    Click here to upload files.


    Yuri

  • Hi Yuri, 
    I have uploaded an zipped example image.

    Thanks for your help
    fsup
  • Hi Yuri,

    I tested your solution. All images are closed. I repeated the test over night with a higher number of passes (noOfLoops = 200). But the process slows down:

    Process started: 02.09.2019 07:27:41
    Loop: 1, CollectionImagesCount: 0, AllImagesCount: 0, Runtime(s): 77,978
    ..
    Loop: 199, CollectionImagesCount: 0, AllImagesCount: 0, Runtime(s): 120,821
    Loop: 200, CollectionImagesCount: 0, AllImagesCount: 0, Runtime(s): 119,93
    Process finished: 02.09.2019 13:20:38

    What could be the cause? How can I fix this?

    Thank you in advance
    fsup
  • Hi Yuri,

    thank you for supporting me. First tests looking very good.

    fsup
Sign In or Register to comment.