Strange issue with privateQueue managedObjectContext

Question

SETUP (You can read this later and skip to the scenario section first)

It's an old app, with manually setup CoreData stack like this:

+ (NSManagedObjectContext *)masterManagedObjectContext
{
    if (_masterManagedObjectContext) {
        return _masterManagedObjectContext;
    }

    NSPersistentStoreCoordinator *coordinator = [self createPersistentStoreCoordinator];

    if (coordinator != nil) {
        _masterManagedObjectContext = [[NSManagedObjectContext alloc] initWithConcurrencyType:NSPrivateQueueConcurrencyType];
        _masterManagedObjectContext.retainsRegisteredObjects = YES;
        _masterManagedObjectContext.mergePolicy = NSOverwriteMergePolicy;
        _masterManagedObjectContext.persistentStoreCoordinator = coordinator;
    }
    return _masterManagedObjectContext;
}

+ (NSManagedObjectContext *)managedObjectContext
{
    if (_managedObjectContext) {
        return _managedObjectContext;
    }

    NSManagedObjectContext *masterContext = [self masterManagedObjectContext];

    if (masterContext) {
        _managedObjectContext = [[NSManagedObjectContext alloc] initWithConcurrencyType:NSMainQueueConcurrencyType];
        _managedObjectContext.retainsRegisteredObjects = YES;
        _managedObjectContext.mergePolicy = NSOverwriteMergePolicy;
        _managedObjectContext.parentContext = masterContext;
    }

    return _managedObjectContext;
}

+ (NSManagedObjectContext *)newManagedObjectContext
{
    __block NSManagedObjectContext *newContext = nil;
    NSManagedObjectContext *parentContext = [self managedObjectContext];

    if (parentContext) {
        newContext = [[NSManagedObjectContext alloc] initWithConcurrencyType:NSPrivateQueueConcurrencyType];
        newContext.parentContext = parentContext;
    }

    return newContext;
}

And then save context recursively:

+ (void)saveContext:(NSManagedObjectContext *)context
{
    [context performBlockAndWait:^{
        if (context.hasChanges && context.persistentStoreCoordinator.persistentStores.count) {
            NSError *error = nil;

            if ([context save:&error]) {
                NSLog(@"saved context: %@", context);

                // Recursive save parent context.
                if (context.parentContext) [self saveContext:context.parentContext];
            }
            else {
                // do some real error handling
                NSLog(@"Could not save master context due to %@", error);
            }
        }
    }];
}

SCENARIO

The app load lots of data from a server, then perform update inside newContext first, then merge into mainContext -> masterContext -> persistentStore.

Because lots of data, the sync process has been divided into about 10 async threads => we have 10 newContext at a time.

Now, the data is complicated, with things like parents <-> children (same class). 1 parent can have many children, and a child can have a mother, father, god father, step mother..., so it's n-n relationship. First, we fetch parent, then perform fetch child and then set the child to parent, and so on.

The server is kinda stupid, it can't send disabled objects. However the customer would like to control the display of app's objects from the back end, so I have 2 properties to do that:

1. hasUpdated: At the beginning of loading process, perform a batch update, set all object's hasUpdated to NO. When got data from the server, update this property to YES.
2. isActive: When all loading was done, perform batch update this property to NO if hasUpdate == NO. Then, I have a filter that won't show object with isActive == NO

ISSUE

Customers complain why some objects being missing even if they're enable in the backend. I've struggle and debugging for so long after got to this strange issue:

1. newContext.updatedObjects : { obj1.ID = 100, hasUpdated == YES }
2. "saved newContext"
3. mainContext.updatedObjects: {obj1.ID = 100, hasUpdated == NO }

// I'll stop here. Obviously, master got updated = NO and finally isActive will set to no, which cause missing objects.

If it happened every time, then probably easier to fix (¿maybe?). However, it occurs like this:

• First time running (by first time, I mean app start from where appDidFinishLaunch... got called): all correct
• 2nd time: missing (153 objects)
• 3rd time: all correct
• 4th time: missing (153 objects) (again? exactly those with multiple parents, I believe so!)
• 5th time: correct again
• ... so on.

Also, it looks like this happened for objects which have the same context (same newContext). Unbelievable.

QUESTIONS

Why is this happening? How do I fix this? If those objects don't have children, my life would be easier!!!!

BONUS

In case you'd like to know how the batch update is, it's below. Note:

1. Download requests are in async queue: _shareInstance.apiQueue = dispatch_queue_create("product_request_queue", DISPATCH_QUEUE_CONCURRENT);
2. Parse response and update properties are syncronous in a queue: _shareInstance.saveQueue = dispatch_queue_create("product_save_queue", DISPATCH_QUEUE_SERIAL);
3. Whenever parse complete, I perform save newContext and call for updateProductActiveStatus: in the same serial queue. If all requests are finished, then perform batch update status. Since request are done in concurent queue, it's always finished earlier than save (serial) queue, so it's pretty much fool proof process.

Code:

// Load Manager
- (void)resetProductUpdatedStatus
{
    NSBatchUpdateRequest *request = [NSBatchUpdateRequest batchUpdateRequestWithEntityName:NSStringFromClass([Product class])];
    request.propertiesToUpdate = @{ @"hasUpdated" : @(NO) };
    request.resultType = NSUpdatedObjectsCountResultType;

    NSBatchUpdateResult *result = (NSBatchUpdateResult *)[self.masterContext executeRequest:request error:nil];

    NSLog(@"Batch update hasUpdated: %@", result.result);

    [self.masterContext performBlockAndWait:^{
        [self.masterContext refreshAllObjects];

        [[CoreDataUtil managedObjectContext] performBlockAndWait:^{
            [[CoreDataUtil managedObjectContext] refreshAllObjects];
        }];
    }];
}

- (void)updateProductActiveStatus:(SyncComplete)callback
{
    if (self.apiRequestList.count) return;

    NSBatchUpdateRequest *request = [NSBatchUpdateRequest batchUpdateRequestWithEntityName:NSStringFromClass([Product class])];
    request.predicate = [NSPredicate predicateWithFormat:@"hasUpdated = NO AND isActive = YES"];
    request.propertiesToUpdate = @{ @"isActive" : @(NO) };
    request.resultType = NSUpdatedObjectsCountResultType;

    NSBatchUpdateResult *result = (NSBatchUpdateResult *)[self.masterContext executeRequest:request error:nil];
    NSLog(@"Batch update isActive: %@", result.result);

    [self.masterContext performBlockAndWait:^{
        [self.masterContext refreshAllObjects];

        NSManagedObjectContext *maincontext = [CoreDataUtil managedObjectContext];
        NSLog(@"Refreshed master");

        [maincontext performBlockAndWait:^{
            [maincontext refreshAllObjects];

            NSLog(@"Refreshed main");

            // Callback
            if (callback) dispatch_async(dispatch_get_main_queue(), ^{ callback(YES, nil); });
        }];
    }];
}

Answer 1

mergePolicy is evil. The only correct mergePolicy is NSErrorMergePolicy any other policy is asking core-data to silently fail and not update when you expect it too.

I suspect that your problem is that you are writing simultaneously to core-data with the background contexts. (I know that you say you have a serial queue - but if you call performBlock inside the queue then each block is executed simultaneously). When there is a conflict stuff gets overwritten. You should only write to core-data in one synchronous way.

I wrote an answer on how to accomplish this with a NSPersistentContainer: NSPersistentContainer concurrency for saving to core data and I would suggest that you migrate your code to it. It really should not be that hard.

If you want to keep the code as close to what is currently is as possible that also is not that hard.

Make a serial operation queue:

_persistentContainerQueue = [[NSOperationQueue alloc] init];
_persistentContainerQueue.maxConcurrentOperationCount = 1;

And do all writing using this queue:

- (void)enqueueCoreDataBlock:(void (^)(NSManagedObjectContext* context))block{
    void (^blockCopy)(NSManagedObjectContext*) = [block copy];

    [self.persistentContainerQueue addOperation:[NSBlockOperation blockOperationWithBlock:^{
        NSManagedObjectContext* context =  [CoreDataUtil newManagedObjectContext];
        [context performBlockAndWait:^{
            blockCopy(context);
            [CoreDataUtil saveContext:context];
        }];
    }]];
}

Also it could be that the objects ARE updated, but you aren't seeing it because you are relying on a fetchedResultsController to be updated. And fetchedResultsController don't update from batch update requests.