Is it possible to do a project-reference the Nuget project (B) from the Solution (A) when building Debug. And when building Release use the Nuget package from Source?
Certainly, but there are some restrictions you need to know.
First, the ID of the NuGet package should different from the name of the reference project, otherwise, the reference from NuGet will replace the project reference.(For example, TestProjectReferenceForDebug is the name of the project reference, if you want to use project reference and NuGet package at the same time, you could not use this project to create the NuGet package directly, so I created a same project with different name to create the NuGet package "TestNuGetForRelease"):

Second, you should use Condition attribute in the ItemGroup element, otherwise, there is an ambiguous reference between 'TestProjectReferenceForDebug' and 'TestNuGetForRelease', so we need add the Condition attribute in the ItemGroup element
<ItemGroup Condition=" '$(Configuration)|$(Platform)' == 'Release|AnyCPU'">
<Reference Include="TestNuGetForRelease, Version=1.0.0.0, Culture=neutral, processorArchitecture=MSIL" >
<HintPath>..\packages\TestNuGetForRelease.1.0.0\lib\net462\TestNuGetForRelease.dll</HintPath>
<Private>True</Private>
</Reference>
</ItemGroup>
<ItemGroup Condition=" '$(Configuration)|$(Platform)' == 'Debug|AnyCPU'">
<ProjectReference Include="..\TestProjectReferenceForDebug\TestProjectReferenceForDebug.csproj">
<Project>{90424b17-2231-4d7d-997b-608115d9f4d9}</Project>
<Name>TestProjectReferenceForDebug</Name>
</ProjectReference>
</ItemGroup>
Third, after we add the Condition attribute in the ItemGroup element with debug and release, we could use project reference in Debug and Nuget in Release, however, if we use those namespace in one .cs file at same time, we need to add those two namespace, then you will get an error "The referenced component 'xxx' could not be found". That because VS could not find those two namespace only in the "Release" or "Debug" model:

To resolve this error, we have to annotate the namespace which in another configuration model when you change the configuration model from Debug to Release.
