FEATURE MANAGER ISSUE with VERY LARGE IMAGE and LARGE NUMBER OF FEATURES in PREMIER 9.3 (3D) . . .

mbatchelor

2017-04-06-083348

All --

I am working on a project that is working with very large images generated by a SCANNER.

The example shown in the SCREEN CAPTURES below is 6074 x 6778.

GOOD NEWS . . .

I am able. to create a SMART THRESHOLD which does a nice job of identifying the BROWN STAINED CELLS in this image.  This is illustrated by IMAGE 1 and IMAGE 2 below where the ORIGINAL IMAGE is on the LEFT and the MASK generated by the SMART THRESHOLD TOOL is on the RIGHT.

After PREMIER 9.3 (3D) performs a COUNT (with no ROI) there are 251707 FEATURES.  This is shown in IMAGE 3 and IMAGE 4 below.

BAD NEWS . . .

This number of FEATURES seems to overwhelm the FEATURE MANAGER.  This is shown in IMAGE 5 where the ADD ALL is DISABLED.

When I put a 1000 x 1000 ROI into the image, the FEATURE MANAGER sees the ROI but does not see the 200K FEATURES.  This is shown in IMAGE 6.

GOOD NEWS . . .

When I run the COUNT within the ROI, the FEATURE MANGER sees the ROI and the 4000 FEATURES.  This is shown in IMAGE 7 and IMAGE 8.

BAD NEWS . . .

When the ROI is increased (here to about 2500 x 2500) and the number of FEATURES is 30,000, the FEATURE MANAGER cannot perform its job.  This is shown in IMAGE 9.

QUESTION . . .

Is there option or configuration within PREMIER that will allow the FEATURE MANAGER to handle the large number of FEATURES that can be generated by LARGE IMAGES like this one (example image attached last below)?

NOTE . . .

The MEMORY LIMIT SETTING within PREMIER is already at the MAXIMUM.  This is shown in IMAGE 10.  This was done and PREMIER was RESTARTED with this setting before the example documented here was performed.

Thanks.

-- Matt

*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-

YuriG

Hi Matt,

Premier uses a fast way to display large number of features, showing them as a single bitmap, if the number of features is too large (>10000 by default). Displaying individual features one by one slows down the update and other UI operations.

You can increase the limit changing "Individual object display limit" on the Application tab of the application options. 

You may also think about other, faster, ways of dealing with large number of objects, like using a mask.

Yuri

mbatchelor

2017-04-06-093110

All --

I forgot to attach the SMART THRESHOLD SEGMENTATION FILE for this image and these FEATURES.

Here it is as:

    SCC Threshold Brown.isg

OOPS . . .

The FORUM does not seem to want me to attach an ISG FILE.  I have renamed this file to be a TXT FILE.

If the FORUM does not mangle it somehow, it should be usable after downloading and renaming.

Thanks.

-- Matt

mbatchelor

2017-04-06-100715

Yuri --

Thank you for your quick response.

I adjusted the OPTION you directed me to to be 1 million and restarted PREMIER.

GOOD NEWS . . .

PREMIER and its FEATURE MANAGER are now able to handle all of the 200,000 + FEATURES found within this image by the SMART SEGMENTATION.  This can be seen in the SCREEN CAPTURE shown below.

BAD NEWS . . .

Unfortunately on my 2013 laptop with 12 GB of RAM and not the fastest processor on the planet, the response time of PREMIER is VERY SLOW.

Can you be a bit more specific about your suggestion about using a MASK?  The END USER is looking to know MEASUREMENT DATA that I do not think can be extracted from a MASK without doing another THRESHOLD and COUNT.  They would also like to be able to DELETE the FALSE POSITIVES and ADD FEATURES that were missed.

Thanks again.

-- Matt

CraigW

You are getting into sizes of measurement features that are pushing the limits of practicality here with the current design of Premier.  But I have a few suggestions:

• You should definitely NOT set the Options->Application->Memory Management->Memory Limit to maximum.  This limit gets the amount of memory that is set aside for image storage, but measurement features and display uses non-image-storage memory.  You should probably leave the Memory Limit at the default of 60% though the optimum may be 50% or 70% for your particular task.  Allocating less image memory will slow down the actual image access for very large images (since some swapping to disk may be required), but this is vastly preferable to getting into a situation where the operating system memory swapping starts to happen.
• The individual display limit only affects how the graphic overlay for detected features is handled, not the underlying measurement feature.  You can still select the measurement feature in the data table, but the corresponding feature in the image will not be highlighted.
• You should consider ways of limiting the number of features that need to be dealt with by careful segmentation and filtering.  In normal circumstances many of the 200,000 detected features will be artifacts and can be rejected from the measurements via filtering (e.g., filter out very small objects).
• Finally, with very large numbers of features, the user cannot reasonably examine individual features, so you need to rely on statistics for results.

YuriG

Matt,

As you found out displaying 1 million features as individual objects requires a lot of memory and can be slow, that's why we have such optimization.
As Craig mentioned, the display limit just affects the presentation on the image overlay, not the results, you can get measurement results of all 1 million features in the data table and calculate statistics.(it would be advised to collect only statistics to Data Collector, if you are going to process multiple images).
Also, pointed by Craig, examining and editing of 1 million features manually is not practically feasible, there will always be features missed from the manual examination, so I would recommend using filter ranges on various parameters to be sure that the outliers are automatically filtered out.

Yuri

CraigW

Also, the Features Manager is not appropriate for such huge numbers of features, and it will only work if your image has fewer features than the individual display limit because it tries to save the appearance of the feature as well as its coordinate information (that's actually probably a bug, since it should either complain or else ignore individual graphic overlay information if it is not present; I'll look into it).  Why do you want to put these measurement features into the Feature Manager in any case?  If your goal is to convert some measured features into a ROI (via the Feature Manager), then clearly you would want to filter out the vast majority of your 200,000 features before doing so.

If all you want is measurement information, then the Measurement Data Table and the Data Collector are the places to look; there is no need to put the measurements into the Features Manager.

mbatchelor

2017-04-06-132835

Craig and Yuri --

The message below was started about 1.5 hours ago and I was working through addressing the points in both of your messages and my system bogged down so much that I had to reboot.

I think some of the info below addresses Craig's question about why I'm using the FEATURE MANAGER in the way I am.

Additionally, PREMIER was booting VERY SLOWLY and then not ever giving me control even though there were no images open and only one project loaded but not visible.  I found that there was a

    FeaturesManager.roi

file created today that was 66 MB in size.  I killed the PREMIER PROCESS, deleted that ROI file and PREMIER seemed to be much less constipated.

I set the MEMORY SETTING to 1/2 on scale, restarted and PREMIER took about 300 MB of MEMORY per the TASK MANAGER.  I opened the example image and premier took about 600 MB or MEMORY.  I recalled the ROI FILE that I saved for the whole image (also about 66 MB) and PREMIER took about 2.8 GB of MEMORY.  When I cleared the FEATURES from the FEATURE MANAGER, PREMIER went back to about 600 MB of MEMORY per the TASK MANAGER.

I think there may be an issue in there someplace.

I'm late for lunch (130pm here now) and I have a phone meeting scheduled at 3pm so I'm going to press POST without dotting all of the Is and crossing all of the Ts.

Thank you for your assistance and I hope the information I have provided is helpful.

-- Matt

*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-

2017-04-06-120209

Craig and Yuri --

Thank you for your responses and the information they contain.

This QUESTION / DISCUSSION was generated because a the FEATURE MANAGER was not operating properly when I tried to use it on LARGE IMAGES with LARGE NUMBERS of COUNTED OBJECTS.

The FEATURE MANAGER and its ability to create ROI FILES was made part of this project because there are two SMART THRESHOLDS being applied to the image and it seemed prudent to create a mechanism that could

** find the two sets of OJBECTS with the two SMART THRESHOLDS
** store the OBJECTS separate from the image
** allow the user to recall, review, edit, and resave the two sets of OBJECTS without recounting them

Yuri's suggestion regarding the INDIVIDUAL OBJECT DISPLAY LIMIT seems to have resolved this issue but now there is a speed issue that is probably aggravated by the fact that while LAPTOP is was good in 2013, it is not STATE OF THE ART in 2017.

But . . .

Unless the FEATURE MANAGER is being fed by the GRAPHIC OVERLAY then there is a disconnect between Yuri's remedy and Craig's statement that

"The individual display limit only affects how the graphic overlay for detected features is handled, not the underlying measurement feature.  You can still select the measurement feature in the data table, but the corresponding feature in the image will not be highlighted."


-- Matt

CraigW

Matt, if you just want to set the current measurements as ROIs, here's a little macro that Yuri worked up to do that without having to use the Features Manager:

    Public Sub ConvertMeasurementsToROIs

        If ThisApplication.ActiveImage Is Nothing Then Exit Sub

        Dim md As Mediacy.Addins.Measurements.McMMData=ThisApplication.ActiveImage.MeasurementsData

        If md.SubFeatures.Count=0 Then Exit Sub

        'get McRegions from the first features

        Dim rgns As MediaCy.IQL.Features.McRegions=TryCast(md.SubFeatures(0).GetFeatures,MediaCy.IQL.Features.McRegions)

        If rgns IsNot Nothing Then

            'create ROIs from measurement features

            ThisApplication.ActiveImage.Aoi.CopyFrom(rgns)

        End If

    End Sub