Assuming you are loading your assembly via Assembly.Load() or Assembly.LoadFrom(), then as explained in this answer to SerializationException for dynamically loaded Type by Chris Shain, you can use the AppDomain.AssemblyResolve event to load your dynamic assembly during deserialization.  However, for security reasons you will want to prevent loading of entirely unexpected assemblies.
One possible implementation would be to introduce the following:
public class AssemblyResolver
{
    readonly string assemblyFullPath;
    readonly AssemblyName assemblyName;
    public AssemblyResolver(string assemblyName, string assemblyFullPath)
    {
        // You might want to validate here that assemblyPath really is an absolute not relative path.
        // See e.g. https://stackoverflow.com/questions/5565029/check-if-full-path-given
        this.assemblyFullPath = assemblyFullPath;
        this.assemblyName = new AssemblyName(assemblyName);
    }
    public ResolveEventHandler AssemblyResolve
    {
        get
        {
            return (o, a) =>
                {
                    var name = new AssemblyName(a.Name);
                    if (name.Name == assemblyName.Name) // Check only the name if you want to ignore version.  Otherwise you can just check string equality.
                        return Assembly.LoadFrom(assemblyFullPath);
                    return null;
                };
        }
    }
}
Then, somewhere in startup, add an appropriate ResolveEventHandler to AppDomain.CurrentDomain.AssemblyResolve e.g. as follows:
class Program
{
    const string assemblyFullPath = @"C:\Full-path-to-my-assembly\MyAssembly.dll";
    const string assemblyName = @"MyAssembly, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null";
    static Program()
    {
        AppDomain.CurrentDomain.AssemblyResolve += new AssemblyResolver(assemblyName, assemblyFullPath).AssemblyResolve;
    }
This ResolveEventHandler checks to see whether the requested assembly has your dynamic assembly's name, and if so, loads the current version from the expected full path.
An alternative would be to write a custom SerializationBinder and attach it to BinaryFormatter.Binder.  In BindToType (string assemblyName, string typeName) the binder would need to check for types belonging to your dynamic assembly, and bind to them appropriately.  The trick here is dealing with situations in which your dynamically loaded types are nested in a generic from another assembly, e.g. a List<MyClass>.  In that case assemblyName will be the name of the assembly of List<T> not MyClass.  For details on how to do this see
In comments @sgnsajgon asked, I wonder why I cannot deserialize stream the same way I would do when signed assembly is explicitly referenced in project - just formatter.Deserialize(stream) and nothing else.
While I don't know what Microsoft employees were thinking when they designed these classes (back in .Net 1.1), it might be because:
- In the words of Eric Lippert, no one ever designed, specified, implemented, tested, documented and shipped that feature. 
- BinaryFormattersecurity is already somewhat of a dumpster fire, but automatically calling- Assembly.Load()on any unexpected assembly name in a- BinaryFormatterstream might make things even worse.
 - By "dumpster fire" I mean that, out-of-the-box, - BinaryFormatterwill instantiate and populate the types specified in the input stream which might not be the types you are expecting.  Thus you might do
 -  var instance = (MyClass)new BinaryFormatter().Deserialize(stream);
 - But if the stream actually contains a serialized attack gadget such as - TempFileCollectionthen the gadget will be created and populated and the attack will be effected.
 - (For details on this sort of attack, see TypeNameHandling caution in Newtonsoft Json, External json vulnerable because of Json.Net TypeNameHandling auto?,  How to configure Json.NET to create a vulnerable web API and Alvaro Muñoz & Oleksandr Mirosh's blackhat paper https://www.blackhat.com/docs/us-17/thursday/us-17-Munoz-Friday-The-13th-JSON-Attacks-wp.pdf.  Those links specify how to modify Json.NET's configuration to enable such attacks; - BinaryFormatteris vulnerable to them in default configuration.)
 - Now if - BinaryFormatterwere automatically calling- Assembly.Load()on unrecognized assembly names, apps using it might additionally become vulnerable to a DLL planting attack where attack types from attack DLLs would get unexpectedly loaded from some unexpected location rather than a secure location, further exacerbating the attack risk.
 - (Incidentally, if you do choose to write your own - SerializationBinderyou can filter out unexpected types and/or known attack types, thus reducing the risk of attack gadget injection.  This also can be harder than expected since- BinaryFormatterstreams often include serialized private or internal classes that you may not know to allow.)
 
As an aside, What are the deficiencies of the built-in BinaryFormatter based .Net serialization? gives a useful overview of other problems you might encounter using BinaryFormatter.