Batch processing - Limitations

I have managed to create a macro that makes a few counts, AND operation, count again, serving results in Excel, closing images, bla bla bla. Everything works fine so far. I can even make this work in a batch processing, but now comes the problems.I want to batch process a lot of images (>500) and it seems like the opening/closing of images makes Image-Pro choke. I get a message of exceeded number of GID (???) 9800 and the batch wont go any further.

Have I reach a limit or can I somehow clean up cash as we go along?

 - Torben Mandrup

Best Answer

  • Accepted Answer
    Hi Torben,

    I don't see this problem with StubNo 1 Image 27.png. Can you please attach StubNo 1 Image 245.png, which you use for tests?

    Yuri
«1

Answers

  • Hi Torben,

    You are reaching Windows limitation of the graphical objects (GDI). Several GDI used for each open image. The main reason of reaching this limit then image is not fully released (some references to the image kept in application or macro).

    Could you past your macro?

    Thanks,
    Nikita.
  • Hi Nikita - Thanks for your reply. You're excactly right. I can see on the batch processing task manager, that closing the last window takes longer and longer and longer as macro is running. This is the macro:

    Public Function FD007() As SimpleScript
    FD007 = New SimpleScript
    Dim doc1, image1, doc2
    Dim docList1 = New List(1), doc3, image2, window1, window2, window3

    With Application.RibbonCommands.SelectRibbonTab(FD007)
    .TabName = "CountSize"
    .Run()
    End With

    With Application.DocumentCommands.Active(FD007)
    .Run(doc1)
    End With

    With Measure.MeasurementsCommands.Options(FD007)
    .ActiveClass = 1
    .Run(doc1)
    End With

    With Measure.Measurements.OptionsCommands.Open(FD007)
    .FileName = "C:\Users\torbe\Dropbox\Torben Dokumenter\Mandrup Software\Kunder\RW\Ny SEM\Image-Pro\FD_Oprindelig.iqo"
    .FilterIndex = 1
    .Run(doc1)
    End With

    With Measure.MeasurementsCommands.ExecuteCount(FD007)
    .Run(doc1)
    End With

    With Measure.Measurements.TableCommands.SendToExcel(FD007)
    .UseStatistics = False
    .Run(doc1)
    End With

    With Measure.Measurements.CreateCommands.Mask(FD007)
    .Run(doc1, image1)
    End With

    With Application.DocumentCommands.Activate(FD007)
    .Run(image1, doc2)
    End With

    With Application.DocumentCommands.Open(FD007)
    .Filenames = New String() {"C:\Users\torbe\Dropbox\Torben Dokumenter\Mandrup Software\Kunder\RW\Ny SEM\Procesbilleder\FiveLines.bmp"}
    .Run(docList1)
    End With

    With Application.DocumentCommands.Activate(FD007)
    .Run(docList1(0), doc3)
    End With

    With Application.RibbonCommands.SelectRibbonTab(FD007)
    .TabName = "Process"
    .Run()
    End With

    With Process.ProcessCommands.Operations(FD007)
    .Operation = eOperation.AND
    .NumberValue = 0R
    .ResultType = eResultType.Apply
    .Run(doc3, image1, image2)
    End With

    With Application.RibbonCommands.SelectRibbonTab(FD007)
    .TabName = "CountSize"
    .Run()
    End With

    With Measure.MeasurementsCommands.Options(FD007)
    .ActiveClass = 1
    .Run(doc3)
    End With

    With Measure.Measurements.OptionsCommands.Open(FD007)
    .FileName = "C:\Users\torbe\Dropbox\Torben Dokumenter\Mandrup Software\Kunder\RW\Ny SEM\Image-Pro\FD_Stumper.iqo"
    .FilterIndex = 1
    .Run(doc3)
    End With

    With Measure.MeasurementsCommands.ExecuteCount(FD007)
    .Run(doc3)
    End With

    With Measure.Measurements.TableCommands.SendToExcel(FD007)
    .UseStatistics = False
    .Run(doc3)
    End With

    With Measure.MeasurementsCommands.DeleteAll(FD007)
    .Run(doc3)
    End With

    With Application.WindowCommands.Define(FD007)
    .Run(doc3, window1)
    End With

    With Application.WindowCommands.Close(FD007)
    .Run(window1)
    End With

    With Application.DocumentCommands.Activate(FD007)
    .Run(doc2, doc2)
    End With

    With Application.WindowCommands.Define(FD007)
    .Run(doc2, window2)
    End With

    With Application.WindowCommands.Close(FD007)
    .Run(window2)
    End With

    With Application.DocumentCommands.Activate(FD007)
    .Run(doc1, doc1)
    End With

    With Measure.MeasurementsCommands.DeleteAll(FD007)
    .Run(doc1)
    End With

    With Application.WindowCommands.Define(FD007)
    .Run(doc1, window3)
    End With

    With Application.WindowCommands.Close(FD007)
    .Run(window3)
    End With

    End Function

    End Module
  • I just ran the macro on 317 images - The first one takes 5 sec and the last one 50 sec. Big problem, but what to do about it?
  • Hi Torben,

    Your macro looks ok, but to reproduce the problem we will need your option files. Can you please zip all OPT files you use in the macro, FiveLines.bmp and at least one sample image and attach it to your post.

    Also, what version of IP Premier do you use? (The latest version is 9.2).

    Thanks,

    Yuri

  • I am relatively new on this:- ) Yes I am using 9.2 but I have no idea what is OPT files - and where to find them!?
    I have attached sample file, FiveLines and iqo files, but I am not sure that's what you're looking for?

     - Torben
  • Hi Tobren,

    Thanks for the provided code and sample images. We were able to reproduce the problem and will work to fix it for the next update.

    Meanwhile I can give you some recommendations to make processing faster:
    - try to work on invisible images - it will significantly speed up the processing and avoid resource leak.
    - in your current macro you export 2 different data tables into Excel for every image, so in the end of batch processing you have a very large Excel sheet with mix of data tables (that large Excel sheet can be the reason of longer processing in the end of the batch). I would recommend to use Data Collector for that purpose, it will give you more structural view of the results (data collector may include several data tables, if necessary). You can also collect only statistics, not data from every segmented objects, if it will work for you.

    Best regards,

    Yuri

  • Hi Yuri - Thank's for your reply. I am already working with the data collector instead - much better:-))

    But when I disable 'Display Documents' in the batchprocessing, nothing happens - the macro is not executed. So what do you mean working on invisible images!?

    Regars Torben
  • Torben,

    Running macro on hidden images requires some modifications to the recorded macro:

    1. You add a parameter to the macro that defines the document loaded by batch processing, like this:
        Public Function LoopOnHidden(doc1 As Object) As SimpleScript
    
    2. be sure to remove doc1 from the list of internal variables, so doc1 is the document ID that will be processed by the macro.

    3. If you have functions inside the macro that load or create new images, add .Visible=false property, so the new image will be loaded or created without showing it:

            With Measure.Measurements.CreateCommands.Mask(LoopOnHidden)
                'create invisible image
                .Visible=False
                .Run(doc1, image1)
            End With
    
            With Application.DocumentCommands.Open(LoopOnHidden)
                .Filenames = New String() {ThisApplication.Path(mcPathType.mcptConfigurationFiles) & "FiveLines.bmp"}
                'open invisible image
                .Visible=False
                .Run(docList1)
            End With
    4. Collect data to Data Collector instead of saving data tables (data tables handle only visible images). Use doc ID to collect the data, like this:
            'collect data
            With Measure.Data.CollectorCommands.Collect(LoopOnHidden)
                .Run(New List({doc1}))
            End With
    
    5. Close all images in the end of the macro using CodeCommand:

            With Automate.ScriptingCommands.CodeCommand(LoopOnHidden)
                If .Run() Then
                    ' User Code Here
                    ' close all images, avoid user prompt clearing Modified flag
                    doc3.Data.Modified=False
                    doc3.Data.Close
                    doc1.Data.Modified=False
                    doc1.Data.Close
                    image1.Modified=False
                    image1.Close
                    ThisApplication.Documents.CloseAll
                End If
            End With

    I've attached the complete project. Note, that you will have to setup proper measurements to collect for Data Collector, you can either do it manually before running the batch or in a macro that is set as "Bun Before" in batch dialog.

    Run the batch with "Display Documents" off.

    Regards,

    Yuri





  • Thanks a lot - I will try this tomorrow:-)))
  • It works - in theory - and is faster by the way:-). But this is VERY strange, because it gives me different results in the datacollector depending on the visibility or not. How the heck is that possible!?

    I have attached a comparison of the two measurements. But the strangest thing is, that if run your macro and set visibility=True I get the same results (the right ones) as with the original macro.

     - Torben

  • edited November 2016
    Hi Torben,

    I checked it and found that loading IQO options from file doesn't set the threshold if the image is not visible, so I've added a line that sets threshold explicitly after loading IQO (the same threshold as in IQO file):

            With Measure.Measurements.OptionsCommands.Open(LoopOnHidden)
                .FileName = ThisApplication.Path(mcPathType.mcptConfigurationFiles) & "FD_Oprindelig.iqo"
                .FilterIndex = 1
                .Run(doc1)
            End With
    
            With Measure.ThresholdToolCommands.Thresholds(LoopOnHidden)
                .AllowOverlap = False
                .Interpretation = eInterpretation.Mono
                .Classes = New System.Collections.Generic.List(Of SegmentationClass)
                .Classes.Add(New SegmentationClass("Class 1",System.Drawing.Color.Blue,New Double(){139R,255R}))
                .Run(doc1)
            End With
    

    I've attached a new version of the project.

    Regards,

    Yuri
  • You're a wizard:-) Now I get similar results for the original image but for processed Fivelines_AND_Mask image there still the difference:-(

     - Torben
  • ....but it's great - it's REALLY fast...three times fast appr
  • All the "Line:End Y" must be 0,5,10,15,20 be definition. Because they are AND'ed from the five lines.

     - Torben
  • Hi Tobren,

    I tried your image 245, but I get correct results:


    Please try to run the batch from my project (without any editing), making LoopOnHidden as the batch macro and processing a folder with one image 245 to check if you have the same results.

    Yuri
  • That is pretty weird - It works. I get slightly different result in the end, but I will look into that. That may be my processing of numbers...
     - Torben
  • Hmmm, there is obviously something I don't undertand about projects here. Does your module1.vb exists as a seperate file (I cannot find it) or is it somehow embedded into the project file?
     - Torben
  • Torben,

    IPX file contains all project files including Module1.vb (it's a ZIP). That's the best way redistributing Premier macro projects.

    Yuri

  • Nice - But how can I unwrap it - If I want to change for instance the configuration files (threshold etc) - Is there a best practice tutorial some where?
  • edited November 2016
    Torben,

    You can unwrap the project file in the Project Workbench using File | Save | Unpack Project Package.
    You can find more info in Project Workbench help (accessible from the help button dropdown of the Project workbench).

    There are also several tutorials on this page http://www.mediacy.com/imagepropremier/learn

    Yuri



  • This is great - Everything seems to work fine. I would like to use the option of monitoring a folder (I have a SEM microscope feeding images into). Is there something in this macro that disables that option?
     - Torben
  • Hi Torben,

    Monitor Folders option is available for any macro, it will process images as soon as they appear in the folder.

    Yuri

  • Hmmm doesn't seem to work right - Is there an settings I should be aware of?
  • edited November 2016
    No, there is no additional options. Select the folder, activate the Monitor Folders checkbox and click the Start button. Then copy a file to your folder - it should be processed by the LoopOn macro.
    I can confirm that LoopOnHidden works with that option.

    Yuri

  • edited January 2017

    Sorry - To bother you again Yuri. I can manage to unwrap project files into another folder. But how do I get it back. I simply can't make it work, it's like your code is hidden in a box I can't get into!?

    I have tried to make a new module, but it gives a different result than yours (LoopOnHidden) - I cannot figure out why.

     - Torben

  • I may be a similar thing as for the threshold setting. I am trying to change the ranges when I do the counting, perhaps that is not loaded either when working on hidden images?

     - Torben

  • Hi Torben,

    You can "unwrap" IPX project using the Unpack Project package menu item in the Project workbench menu:



    Threshold tool recording also works with visible images, so you have to modify the macro to make it work on invisible.

    Yuri
  • 2017-08-02-152044

    Yuri --

    If an IMAGE is OPEN and VISIBLE, can it be made INVISIBLE?

    If YES, can you please provide a ROUTINE to do this?

    My attempt is below but it seems to CLOSE the VISIBLE IMAGE rather than making it INVISIBLE.

    If NO, I guess the OPEN IMAGE must be SAVED, CLOSED, and then OPENED (as INVISIBLE).

    Thanks.

    -- Matt

    *-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-
        Public Function Make_Image_Invisible2()
    
            Debug.Print "ThisApplication.Documents.Values.Count"
            Debug.Print ThisApplication.Documents.Values.Count
    
            ThisApplication.ActiveImage.Visible = False
    
            Debug.Print "ThisApplication.Documents.Values.Count"
            Debug.Print ThisApplication.Documents.Values.Count
    
        End Function


Sign In or Register to comment.