The issue is more complicated as thought.
I'm going to attach a sample pax exam project for reproduction.
The root of the problem is that pax exam closes the ServiceTracker in the ServiceLookup class. Since that invocation ungets the services on default, see
the method org.osgi.util.tracker.ServiceTracker.removedService(ServiceReference, Object) (line 486)
within the ServiceTracker implementation.
The configuration admin of felix then disposes the service instance. See here
org.apache.felix.cm.impl.ConfigurationAdminFactory.ungetService(Bundle, ServiceRegistration, Object) (line 61)
The implementation of the ConfigurationAdmin of felix seems correct to me since the ConfigurationAdmin service is explicitly ungetted and should be disposed as done. I guess the correct solution is to remove the finally block which closes the service tracker.