I never used the NativeHandleProperty, it was only a possible solution as far as I know which options you have.
I'm not sure why there is an exception, maybe the microsoft forum which handles UIAutomation can help here.
in your solution it seems that the application and the test runs in the same process => this is not supported.
Maybe it would be better to try something like this:
all elements you have problems with - nested or not- are panes, right?
so your start point is a window?
Window window = application.GetWindows().Where(p=>p.Name =="a name").First();
AutomationElementCollection coll = window.AutomationElement.FindAll(TreeScope.Children, new PropertyCondition( AutomationElement.ControlTypeProperty, ControlType.Pane ));
TreeScope.Children => you get only the direct children of the window element.
Maybe next step
List<UIItem> items = new List<UIItem>();
foreach(AutomationElement element in coll)
items.Add(new UIItem(element, window.AutomationElement));
on each UIItem you have, you can now do the same to get the nested pane's
maybe you can identify panes because of specific labels, buttons, or something like that, so I would suggest to write a helper class which returns you the searched pain.
if you don't want to use UISpy all the time you can use LogStructure on each white element.
This will give you all nested elements.