Batch processing - Limitations
I have managed to create a macro that makes a few counts, AND operation, count again, serving results in Excel, closing images, bla bla bla. Everything works fine so far. I can even make this work in a batch processing, but now comes the problems.I want to batch process a lot of images (>500) and it seems like the opening/closing of images makes Image-Pro choke. I get a message of exceeded number of GID (???) 9800 and the batch wont go any further.
Have I reach a limit or can I somehow clean up cash as we go along?
- Torben Mandrup
Have I reach a limit or can I somehow clean up cash as we go along?
- Torben Mandrup
0
Best Answer
-
Hi Torben,
I don't see this problem with StubNo 1 Image 27.png. Can you please attach StubNo 1 Image 245.png, which you use for tests?
Yuri0
Answers
-
Hi Torben,
You are reaching Windows limitation of the graphical objects (GDI). Several GDI used for each open image. The main reason of reaching this limit then image is not fully released (some references to the image kept in application or macro).
Could you past your macro?
Thanks,
Nikita.0 -
Hi Nikita - Thanks for your reply. You're excactly right. I can see on the batch processing task manager, that closing the last window takes longer and longer and longer as macro is running. This is the macro:Public Function FD007() As SimpleScriptFD007 = New SimpleScriptDim doc1, image1, doc2Dim docList1 = New List(1), doc3, image2, window1, window2, window3With Application.RibbonCommands.SelectRibbonTab(FD007).TabName = "CountSize".Run()End WithWith Application.DocumentCommands.Active(FD007).Run(doc1)End WithWith Measure.MeasurementsCommands.Options(FD007).ActiveClass = 1.Run(doc1)End WithWith Measure.Measurements.OptionsCommands.Open(FD007).FileName = "C:\Users\torbe\Dropbox\Torben Dokumenter\Mandrup Software\Kunder\RW\Ny SEM\Image-Pro\FD_Oprindelig.iqo".FilterIndex = 1.Run(doc1)End WithWith Measure.MeasurementsCommands.ExecuteCount(FD007).Run(doc1)End WithWith Measure.Measurements.TableCommands.SendToExcel(FD007).UseStatistics = False.Run(doc1)End WithWith Measure.Measurements.CreateCommands.Mask(FD007).Run(doc1, image1)End WithWith Application.DocumentCommands.Activate(FD007).Run(image1, doc2)End WithWith Application.DocumentCommands.Open(FD007).Filenames = New String() {"C:\Users\torbe\Dropbox\Torben Dokumenter\Mandrup Software\Kunder\RW\Ny SEM\Procesbilleder\FiveLines.bmp"}.Run(docList1)End WithWith Application.DocumentCommands.Activate(FD007).Run(docList1(0), doc3)End WithWith Application.RibbonCommands.SelectRibbonTab(FD007).TabName = "Process".Run()End WithWith Process.ProcessCommands.Operations(FD007).Operation = eOperation.AND.NumberValue = 0R.ResultType = eResultType.Apply.Run(doc3, image1, image2)End WithWith Application.RibbonCommands.SelectRibbonTab(FD007).TabName = "CountSize".Run()End WithWith Measure.MeasurementsCommands.Options(FD007).ActiveClass = 1.Run(doc3)End WithWith Measure.Measurements.OptionsCommands.Open(FD007).FileName = "C:\Users\torbe\Dropbox\Torben Dokumenter\Mandrup Software\Kunder\RW\Ny SEM\Image-Pro\FD_Stumper.iqo".FilterIndex = 1.Run(doc3)End WithWith Measure.MeasurementsCommands.ExecuteCount(FD007).Run(doc3)End WithWith Measure.Measurements.TableCommands.SendToExcel(FD007).UseStatistics = False.Run(doc3)End WithWith Measure.MeasurementsCommands.DeleteAll(FD007).Run(doc3)End WithWith Application.WindowCommands.Define(FD007).Run(doc3, window1)End WithWith Application.WindowCommands.Close(FD007).Run(window1)End WithWith Application.DocumentCommands.Activate(FD007).Run(doc2, doc2)End WithWith Application.WindowCommands.Define(FD007).Run(doc2, window2)End WithWith Application.WindowCommands.Close(FD007).Run(window2)End WithWith Application.DocumentCommands.Activate(FD007).Run(doc1, doc1)End WithWith Measure.MeasurementsCommands.DeleteAll(FD007).Run(doc1)End WithWith Application.WindowCommands.Define(FD007).Run(doc1, window3)End WithWith Application.WindowCommands.Close(FD007).Run(window3)End WithEnd FunctionEnd Module0
-
I just ran the macro on 317 images - The first one takes 5 sec and the last one 50 sec. Big problem, but what to do about it?0
-
Hi Torben,
Your macro looks ok, but to reproduce the problem we will need your option files. Can you please zip all OPT files you use in the macro, FiveLines.bmp and at least one sample image and attach it to your post.
Also, what version of IP Premier do you use? (The latest version is 9.2).
Thanks,
Yuri
0 -
I am relatively new on this:- ) Yes I am using 9.2 but I have no idea what is OPT files - and where to find them!?
I have attached sample file, FiveLines and iqo files, but I am not sure that's what you're looking for?
- Torben0 -
Hi Tobren,
Thanks for the provided code and sample images. We were able to reproduce the problem and will work to fix it for the next update.
Meanwhile I can give you some recommendations to make processing faster:
- try to work on invisible images - it will significantly speed up the processing and avoid resource leak.
- in your current macro you export 2 different data tables into Excel for every image, so in the end of batch processing you have a very large Excel sheet with mix of data tables (that large Excel sheet can be the reason of longer processing in the end of the batch). I would recommend to use Data Collector for that purpose, it will give you more structural view of the results (data collector may include several data tables, if necessary). You can also collect only statistics, not data from every segmented objects, if it will work for you.
Best regards,
Yuri
0 -
Hi Yuri - Thank's for your reply. I am already working with the data collector instead - much better:-))
But when I disable 'Display Documents' in the batchprocessing, nothing happens - the macro is not executed. So what do you mean working on invisible images!?
Regars Torben0 -
Torben,
Running macro on hidden images requires some modifications to the recorded macro:
1. You add a parameter to the macro that defines the document loaded by batch processing, like this:Public Function LoopOnHidden(doc1 As Object) As SimpleScript
2. be sure to remove doc1 from the list of internal variables, so doc1 is the document ID that will be processed by the macro.
3. If you have functions inside the macro that load or create new images, add .Visible=false property, so the new image will be loaded or created without showing it:With Measure.Measurements.CreateCommands.Mask(LoopOnHidden) 'create invisible image .Visible=False .Run(doc1, image1) End With With Application.DocumentCommands.Open(LoopOnHidden) .Filenames = New String() {ThisApplication.Path(mcPathType.mcptConfigurationFiles) & "FiveLines.bmp"} 'open invisible image .Visible=False .Run(docList1) End With
4. Collect data to Data Collector instead of saving data tables (data tables handle only visible images). Use doc ID to collect the data, like this:'collect data With Measure.Data.CollectorCommands.Collect(LoopOnHidden) .Run(New List({doc1})) End With
5. Close all images in the end of the macro using CodeCommand:With Automate.ScriptingCommands.CodeCommand(LoopOnHidden) If .Run() Then ' User Code Here ' close all images, avoid user prompt clearing Modified flag doc3.Data.Modified=False doc3.Data.Close doc1.Data.Modified=False doc1.Data.Close image1.Modified=False image1.Close ThisApplication.Documents.CloseAll End If End With
I've attached the complete project. Note, that you will have to setup proper measurements to collect for Data Collector, you can either do it manually before running the batch or in a macro that is set as "Bun Before" in batch dialog.
Run the batch with "Display Documents" off.
Regards,
Yuri
0 -
Thanks a lot - I will try this tomorrow:-)))0
-
It works - in theory - and is faster by the way:-). But this is VERY strange, because it gives me different results in the datacollector depending on the visibility or not. How the heck is that possible!?
I have attached a comparison of the two measurements. But the strangest thing is, that if run your macro and set visibility=True I get the same results (the right ones) as with the original macro.
- Torben0 -
Hi Torben,
I checked it and found that loading IQO options from file doesn't set the threshold if the image is not visible, so I've added a line that sets threshold explicitly after loading IQO (the same threshold as in IQO file):With Measure.Measurements.OptionsCommands.Open(LoopOnHidden) .FileName = ThisApplication.Path(mcPathType.mcptConfigurationFiles) & "FD_Oprindelig.iqo" .FilterIndex = 1 .Run(doc1) End With With Measure.ThresholdToolCommands.Thresholds(LoopOnHidden) .AllowOverlap = False .Interpretation = eInterpretation.Mono .Classes = New System.Collections.Generic.List(Of SegmentationClass) .Classes.Add(New SegmentationClass("Class 1",System.Drawing.Color.Blue,New Double(){139R,255R})) .Run(doc1) End With
I've attached a new version of the project.
Regards,
Yuri0 -
You're a wizard:-) Now I get similar results for the original image but for processed Fivelines_AND_Mask image there still the difference:-(
- Torben0 -
....but it's great - it's REALLY fast...three times fast appr0
-
All the "Line:End Y" must be 0,5,10,15,20 be definition. Because they are AND'ed from the five lines.
- Torben0 -
Hi Tobren,
I tried your image 245, but I get correct results:
Please try to run the batch from my project (without any editing), making LoopOnHidden as the batch macro and processing a folder with one image 245 to check if you have the same results.
Yuri0 -
That is pretty weird - It works. I get slightly different result in the end, but I will look into that. That may be my processing of numbers...
- Torben0 -
Hmmm, there is obviously something I don't undertand about projects here. Does your module1.vb exists as a seperate file (I cannot find it) or is it somehow embedded into the project file?
- Torben0 -
Torben,
IPX file contains all project files including Module1.vb (it's a ZIP). That's the best way redistributing Premier macro projects.
Yuri
0 -
Nice - But how can I unwrap it - If I want to change for instance the configuration files (threshold etc) - Is there a best practice tutorial some where?0
-
Torben,
You can unwrap the project file in the Project Workbench using File | Save | Unpack Project Package.
You can find more info in Project Workbench help (accessible from the help button dropdown of the Project workbench).
There are also several tutorials on this page http://www.mediacy.com/imagepropremier/learn
Yuri
0 -
This is great - Everything seems to work fine. I would like to use the option of monitoring a folder (I have a SEM microscope feeding images into). Is there something in this macro that disables that option?
- Torben0 -
Hi Torben,
Monitor Folders option is available for any macro, it will process images as soon as they appear in the folder.
Yuri
0 -
Hmmm doesn't seem to work right - Is there an settings I should be aware of?0
-
No, there is no additional options. Select the folder, activate the Monitor Folders checkbox and click the Start button. Then copy a file to your folder - it should be processed by the LoopOn macro.
I can confirm that LoopOnHidden works with that option.
Yuri
0 -
Sorry - To bother you again Yuri. I can manage to unwrap project files into another folder. But how do I get it back. I simply can't make it work, it's like your code is hidden in a box I can't get into!?
I have tried to make a new module, but it gives a different result than yours (LoopOnHidden) - I cannot figure out why.
- Torben
0 -
I may be a similar thing as for the threshold setting. I am trying to change the ranges when I do the counting, perhaps that is not loaded either when working on hidden images?
- Torben
0 -
Hi Torben,
You can "unwrap" IPX project using the Unpack Project package menu item in the Project workbench menu:
Threshold tool recording also works with visible images, so you have to modify the macro to make it work on invisible.
Yuri0 -
2017-08-02-152044
Yuri --
If an IMAGE is OPEN and VISIBLE, can it be made INVISIBLE?
If YES, can you please provide a ROUTINE to do this?
My attempt is below but it seems to CLOSE the VISIBLE IMAGE rather than making it INVISIBLE.
If NO, I guess the OPEN IMAGE must be SAVED, CLOSED, and then OPENED (as INVISIBLE).
Thanks.
-- Matt
*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-Public Function Make_Image_Invisible2() Debug.Print "ThisApplication.Documents.Values.Count" Debug.Print ThisApplication.Documents.Values.Count ThisApplication.ActiveImage.Visible = False Debug.Print "ThisApplication.Documents.Values.Count" Debug.Print ThisApplication.Documents.Values.Count End Function
0
Categories
- All Categories
- 961 Image-Pro v9 and higher
- 9 Image-Pro FAQs
- 18 Image-Pro Download & Install
- 448 Image-Pro General Discussions
- 486 Image-Pro Automation (Macros, Apps, Reports)
- 20 AutoQuant Deconvolution
- 2 AutoQuant Download & Install
- 18 AutoQuant General Discussions
- 195 Image-Pro Plus v7 and lower
- 3 Image-Pro Plus Download & Install
- 106 Image-Pro Plus General Discussions
- 86 Image-Pro Plus Automation with Macros
- 19 Legacy Products
- 16 Image-Pro Premier 3D General Discussions
- 26 Image-Pro Insight General Discussions