.ToList on IQueryable causes NullReferenceException when executed in background

Question

My code works fine when executed synchronously, but fails when triggered via QueueBackgroundWorkItem.

UPDATE: the issue seems to be with the Sitefinity dynamic module manager I am using requiring HTTP context within the thread created by QueueBackgroundWorkItem, so this is a Sitefinity specific question. I've found an article that seems to point to a solution: http://www.sitefinity.com/developer-network/forums/sitefinity-sdk/errors-with-managers-when-multi-threading

Relevant portions of my code

ApiController action that starts background job:

// GET: api/Sync/MFP
[HttpGet]
public IHttpActionResult MFP()
{
    HostingEnvironment.QueueBackgroundWorkItem(ct => startMFPSync());
    return Ok("Sync Started!");
}

private void startMFPSync()
{
    var sourcesConfig = Config.Get<SyncSourcesSettingsConfig>();

    MFPApi api = new MFPApi(new MFPConfig
    {
        Url = sourcesConfig.MFPUrl,
        Key = sourcesConfig.MFPKey,
        Password = sourcesConfig.MFPPassword,
    });

    DataSync dataSync = new DataSync(api);
    dataSync.RunSync();
}

DataSync class:

public class DataSync
{
    IDataSource dataSource;

    public DataSync(IDataSource source)
    {
        dataSource = source;
    }

    public void RunSync()
    {
        dataSource.GetResponse();

        List<SyncContent> dataToSync = dataSource.GetDataForSync();
        SFDynamicModuleSync destinationSync = new SFDynamicModuleSync(dataToSync);

        // function call where exception occurs
        destinationSync.CacheModuleData();

        // other sync operations
    }
}

CacheModuleData() function in class below is where exception occurs:

public class SFDynamicModuleSync : IDataDestinationSync
{
    /// <summary>
    /// List of SyncContent objects to sync
    /// </summary>
    private List<SyncContent> dataToSync;

    /// <summary>
    /// Used to store results of CacheModuleData
    /// </summary>
    private List<List<DynamicContent>> modulesItems;

    /// <summary>
    /// Sitefinity dynamic module manager
    /// </summary>
    private DynamicModuleManager dynamicModuleManager;

    /// <summary>
    /// Initializes a new instance of the <see cref="SFDynamicModuleSync"/> class
    /// </summary>
    /// <param name="dataToSync">List of SyncContent objects to sync</param>
    public SFDynamicModuleSync(List<SyncContent> dataToSync)
    {
        this.dataToSync = dataToSync;
        this.modulesItems = new List<List<DynamicContent>>();
        this.dynamicModuleManager = DynamicModuleManager.GetManager();
    }

    /// <summary>
    /// Retrieves all data from dynamic modules and places in modulesItems
    /// </summary>
    public void CacheModuleData()
    {
        foreach (string contentType in this.dataToSync.Select(e => e.ContentTypeName))
        {
            Type type = TypeResolutionService.ResolveType(contentType);

            IQueryable<DynamicContent> moduleItems = this.dynamicModuleManager.GetDataItems(type)
                .Where(i => i.Status == ContentLifecycleStatus.Master);

            if(moduleItems != null)
            {
                // The .ToList() here causes a NullReferenceException when code is triggered by background job
                List<DynamicContent> moduleItemsList = moduleItems.ToList();
                this.modulesItems.Add(moduleItemsList);
            }
        }
    }

    // other sync methods - not included here for abbrevity 
}

Stack trace:

System.NullReferenceException was unhandled by user code
  HResult=-2147467261
  Message=Object reference not set to an instance of an object.
  Source=Unity_ILEmit_DynamicClasses
  StackTrace:
       at DynamicModule.ns.Wrapped_OpenAccessDynamicModuleProvider_81d3fcbe95dd4a47b8c1cb1cc5a692ab.ApplyFilters(IDataItem item)
       at Telerik.Sitefinity.Security.FieldsPermissionsApplierEnumerator`1.Demand(T forItem)
       at Telerik.Sitefinity.Security.PermissionApplierEnumeratorBase`1.MoveNext()
       at Telerik.Sitefinity.Data.Linq.DataItemEnumerator`1.MoveNext()
       at System.Collections.Generic.List`1..ctor(IEnumerable`1 collection)
       at System.Linq.Enumerable.ToList[TSource](IEnumerable`1 source)
       at TeamSI.Sitefinity.DataSync.DataDestinations.SFDynamicModuleSync.CacheModuleData() in C:\Projects\SIEQ\TeamSI.Sitefinity.DataSync\DataDestinations\SFDynamicModuleSync.cs:line 75
       at TeamSI.Sitefinity.DataSync.DataSync.RunSync() in C:\Projects\SIEQ\TeamSI.Sitefinity.DataSync\DataSync.cs:line 28
       at SitefinityWebApp.Mvc.Controllers.SyncController.startMFPSync() in C:\Projects\SIEQ\Main_Site\Mvc\Controllers\SyncController.cs:line 72
       at SitefinityWebApp.Mvc.Controllers.SyncController.<MFP>b__1_0(CancellationToken ct) in C:\Projects\SIEQ\Main_Site\Mvc\Controllers\SyncController.cs:line 52
       at System.Web.Hosting.HostingEnvironment.<>c__DisplayClass91_0.<QueueBackgroundWorkItem>b__0(CancellationToken ct)
       at System.Web.Hosting.BackgroundWorkScheduler.<RunWorkItemImpl>d__7.MoveNext()
  InnerException:

Is there anything I can add that would make the question more understandable or helpful to others? I realize it's a rather specific question, but would like to make it a question that could help others in similar situations. — jmotes, Jan 07 '17 at 19:12
why not just check to see if `moduleItems` is null? Seems like a non-question question. — Hogan, Jan 07 '17 at 19:16
@Hogan, it would be a good idea - but in this specific case it shouldn't be null since there are module items in my database and there are 500+ items in it when not triggered by the background job. — jmotes, Jan 07 '17 at 19:17
All NullReferenceExceptions begin with words *it shouldn't be null since...* — Sergey Berezovskiy, Jan 07 '17 at 19:19
To further clarify, moduleItems is NOT null even when triggered by the background job and when the exception occurs. I updated the question with a null check. — jmotes, Jan 07 '17 at 19:20
@EugenePodskal it's not a dup of that since "moduleItems" is not null when the exception occurs. — jmotes, Jan 07 '17 at 19:24
Did you check the stacktrace of the exception? You need to post it also. Note also that `moduleItems` is an `IEnumerable`. When you call `moduleItems.ToList()` you are actually triggering the `Where` in the previous line which will execute the predicate on all elements. If one of the elements in the list is `null` then you will get the exception. — daramasala, Jan 07 '17 at 19:26
As already stated please post the entire stack trace of the NRE. — Igor, Jan 07 '17 at 19:26
The call to `.ToList()` materializes the query. That's possible why the exception manifests itself at that point. What exactly is the code in `this.dynamicModuleManager.GetDataItems`? — Peter Bons, Jan 07 '17 at 19:26
@PeterBons, that is a API method in the module manager that the Sitefinity CMS provides. I don't have source code for it. It's just used to query data in custom content types. — jmotes, Jan 07 '17 at 19:31
Having no insights in how the `IQueryable` is build will make it hard to debug. Have you checked in this code `.Where(i => i.Status == ContentLifecycleStatus.Master);` that neither `i` or `i.Status` is `null`? If not the problem is probably somewhere in the underlying method. Which you do not have access to. — Peter Bons, Jan 07 '17 at 19:35
@PeterBons I have tried adding a null check for "i" and still got the error. Have not checked i.Status though. — jmotes, Jan 07 '17 at 19:40
@PeterBons since this code works perfectly if I call the function directly (not through a background job) wouldn't the issue be my code though? — jmotes, Jan 07 '17 at 19:41
I was thinking the issue is related to me not knowing how to write code so it works effectively in a separate thread. — jmotes, Jan 07 '17 at 19:42
Is there anything in the code that requires the http context like a request variable or the authenticated user id from the loaded principle? This would not be available running on a different thread other than the thread that hosts the request context and could possible throw an NRE. — Igor, Jan 07 '17 at 19:44
@Igor good point. When I create the dynamicModuleManager instance, I have to give it some parameters so it can set up user-less authentication. That might can only work in the HTTP request "chain". I'll update the code to include the "SFDynamicModuleSync" class constructor where the dynamicModuleManager init code is. — jmotes, Jan 07 '17 at 19:48
@Igor updated question to include class where the CacheModuleData method exists. I was wrong - I don't have to set up authentication to just read from the CMS module data - just have to call the DynamicModuleManager.GetManager method - but that method could require a HTTP context as well. — jmotes, Jan 07 '17 at 20:04
Why didn't you post the whole exception -- what was the inner exception. **why cut that off**? — Hogan, Jan 07 '17 at 23:02
@Hogan That's just what Visual Studio gave me when I clicked the copy to clipboard link. I'll see if I can get more details if my quest to simulate HTTP context within the thread doesn't help. Thanks for much for your help on this! — jmotes, Jan 08 '17 at 00:45
@Hogan is there a way you can remove the duplicate answer tag, or will it go away on it's own? — jmotes, Jan 08 '17 at 03:48
My question was "why does the nullreferenceexception get triggered when my code is ran in the background?". It wasn't a question of what a nullreferenceexception is or how to add a null check, so isn't a duplicate of 4660142 — jmotes, Jan 08 '17 at 06:08
@jmotes -- you can nominate it for re-open, I don't think this should be nominated. This is a normal null reference exception in the context of threading. I'm sorry if you're disappointed to learn that. You question was well written and in the future if you continue to write (or answer) questions as well I'm sure you will have a popular one. — Hogan, Jan 08 '17 at 15:52
Comments are not for extended discussion; this conversation has been [moved to chat](http://chat.stackoverflow.com/rooms/132632/discussion-on-question-by-jmotes-tolist-on-iqueryable-causes-nullreferenceexcep). — Bhargav Rao, Jan 08 '17 at 16:12

jmotes · Accepted Answer · 2017-01-08T03:50:41.057

Turns out this was an issue with the Sitefinity CMS I was querying data from requiring HTTP context, which is of course lost within threads. The NullReferenceException was occuring within the dynamicModuleManager.GetDataItems() method.

With some help from friends and awesome SO contributors including @Igor, @Hogan, @PeterBons and @daramasala who helped me understand the problem I was able to solve it by upping privileges to SF and simulating HttpContext within the thread.

http://www.sitefinity.com/developer-network/forums/sitefinity-sdk/errors-with-managers-when-multi-threading

Only SF documentation outside of the forums I could find on using this method is here: http://docs.sitefinity.com/bug-tracker-create-the-savebug-action

Updated RunSync() method in my DataSync class:

public void RunSync()
{
    SystemManager.RunWithElevatedPrivilegeDelegate worker = new SystemManager.RunWithElevatedPrivilegeDelegate(args => {

        dataSource.GetResponse();
        List<SyncContent> dataToSync = dataSource.GetDataForSync();
        var destinationSync = new SFDynamicModuleSync(dataToSync);

        destinationSync.CacheModuleData();

        // complete sync operations for each content type
        for (int i = 0; i < dataToSync.Count; i++)
        {
            destinationSync.DeleteOldItems(i);
            destinationSync.AddItems(i);
            destinationSync.UpdateItems(i);
        }
    });

    SystemManager.RunWithElevatedPrivilege(worker);
}

.ToList on IQueryable causes NullReferenceException when executed in background

1 Answers1