Specifics of QueryProvider

QueryProvider can’t deal with this:

var result = _context.Humans
                      .Select(x => $"Name: {x.Name}  Age: {x.Age}")
                      .Where(x => x != "")
                      .ToList();

It can’t deal with any sentence using an interpolated string, but it’ll easily deal with this:

var result = _context.Humans
                      .Select(x => "Name " +  x.Name + " Age " + x.Age)
                      .Where(x => x != "")
                      .ToList();

The most painful thing is to fix bugs after turning on ClientEvaluation (exception for client-side calculation), since all Automapper profiles should be strictly analyzed for interpolation. Let’s find out what’s what and propose our solution to the problem.

Fixing things

Interpolation in the Expression Tree is converted like this (this is a result of ExpressionStringBuilder.ExpressionToString method, it skipped some of the nodes but this is OK):

// boxing is required for x.Age
Format("Name:{0} Age:{1}", x.Name, Convert(x.Age, Object)))

Or like this, if there are more than 3 arguments:

Format("Name:{0} Age:{1}", new [] {x.Name, Convert(x.Age, Object)))

We can conclude that the provider simply was not taught to process these cases, but it might be taught to bring these cases with the well known ToString(), processed like this:

((("Name: " + x.Name) + " Age: ") + Convert(x.Age, Object)))

I want to write a Visitor that will follow the Expression Tree (in particular, the MethodCallExpression nodes) and replace the Format method with concatenation. If you’re familiar with expression trees, you know that C# provides its own visitor to bypass the tree — ExpressionVisitor. More info for those interested.

All we need is to overridethe VisitMethodCall method and slightly modify the returned value. The method parameter is of the MethodCallExpression type, containing information on the method itself and the arguments fed to it.

Let’s divide the task into several parts:

Determine that it’s the Format method that came into the VisitMethodCall;
Replace the method with concatenation of strings;
Handle all overloads of the Format method we can have;
Write the extension method to call our visitor.

The first part is simple: the Format method has 4 overloads built in an Expression Tree:

 public static string Format(string format, object arg0)  
 public static string Format(string format, object arg0,object arg1)  
 public static string Format(string format, object arg0,object arg1,object arg2)
 public static string Format(string format, params object[] args)

Let’s extract them, using their MethodInfo reflection:

private IEnumerable<MethodInfo> FormatMethods =>
            typeof(string).GetMethods().Where(x => x.Name.Contains("Format"))

//first three
private IEnumerable<MethodInfo> FormatMethodsWithObjects => 
   FormatMethods
         .Where(x => x.GetParameters()
         .All(xx=> xx.ParameterType == typeof(string) || 
                        xx.ParameterType == typeof(object))); 

//last one
private IEnumerable<MemberInfo> FormatMethodWithArrayParameter => 
   FormatMethods
        .Where(x => x.GetParameters()
                              .Any(xx => xx.ParameterType == typeof(object[])));

Excellent. Now we can determine if the Format method “came in” to MethodCallExpression.

While bypassing the tree in VisitMethodCall, the following methods can come in:

Format with object arguments
Format with object[] argument
Something else entirely.

A bit of custom pattern matching

Since we only have 3 conditions, we can handle them using if, but since we assume we’ll need to expand this method in the future, let’s offload all cases into this data structure:

public class PatternMachingStructure
 {
    public Func<MethodInfo, bool> FilterPredicate { get; set; }
    public Func<MethodCallExpression, IEnumerable<Expression>> 
                                       SelectorArgumentsFunc { get; set; }
    public Func<MethodCallExpression, IEnumerable<Expression>, Expression> 
                                       ReturnFunc { get; set; }
 }

var patternMatchingList = new List<PatternMachingStructure>()

Using FilterPredicate we determine which of the 3 cases we’re dealing with. SelectorArgumentFunc is needed to bring all arguments of the Format method into a unified shape, the ReturnFunc method, which will return the full Expression.

Now let’s replace interpolation with concatenation, and for that we’ll need this method:

private Expression InterpolationToStringConcat(MethodCallExpression node,
            IEnumerable<Expression> formatArguments)
{
  //picking the first argument
  //(example : Format("Name: {0} Age: {1}", x.Name,x.Age) -> 
  //"Name: {0} Age: {1}"
  var formatString = node.Arguments.First();
  // going through the pattern from Format method and choosing every 
  // line between the arguments and pass them to the ExpressionConstant method
  // example:->[Expression.Constant("Name: "),Expression.Constant(" Age: ")]
  var argumentStrings = Regex.Split(formatString.ToString(),RegexPattern)
                             .Select(Expression.Constant);
  // merging them with the formatArguments values
  // example ->[ConstantExpression("Name: "),PropertyExpression(x.Name),
  // ConstantExpression("Age: "),
  // ConvertExpression(PropertyExpression(x.Age), Object)]
  var merge = argumentStrings.Merge(formatArguments, new ExpressionComparer());
  // merging like QueryableProvider merges simple lines concatenation
  // example : -> MethodBinaryExpression 
  //(("Name: " + x.Name) + "Age: " + Convert(PropertyExpression(x.Age),Object))
  var result = merge.Aggregate((acc, cur) =>
                    Expression.Add(acc, cur, StringConcatMethod));
  return result;
 }

InterpolationToStringConcat will be called from the Visitor, hidden behind ReturnFunc:

protected override Expression VisitMethodCall(MethodCallExpression node)
{
  var pattern = patternMatchingList.First(x => x.FilterPredicate(node.Method));
  var arguments = pattern.SelectorArgumentsFunc(node);
  var expression = pattern.ReturnFunc(node, arguments);
  return expression;
}

Now we need to write logic for handling all Format method overloads. It’s rather trivial and is located inside the patternMatchingList:

patternMatchingList = new List<PatternMachingStructure>
{
    // first three Format overloads
   new PatternMachingStructure
   {
        FilterPredicate = x => FormatMethodsWithObjects.Contains(x),
        SelectorArgumentsFunc = x => x.Arguments.Skip(1),
        ReturnFunc = InterpolationToStringConcat
    },
    // last Format overload receiving the array
    new PatternMachingStructure
    {
        FilterPredicate = x => FormatMethodWithArrayParameter.Contains(x),
        SelectorArgumentsFunc = x => ((NewArrayExpression) x.Arguments.Last())
                                                            .Expressions,
        ReturnFunc = InterpolationToStringConcat
     },
     // node.Method != Format
    new PatternMachingStructure()
    {
        FilterPredicate = x => FormatMethods.All(xx => xx != x),
        SelectorArgumentsFunc = x => x.Arguments,
         ReturnFunc = (node, _) => base.VisitMethodCall(node)
     }
};

Accordingly, we’ll follow that list in the VisitMethodCall method until the first positive FilterPredicate, then convert the arguments (SelectorArgumentFunc) and execute ReturnFunc.

Let’s write an extension method which we can call to replace interpolation.

We can get an Expression, hand it off to the Visitor, and then call the CreateQuery method replacing the original Expression Tree with ours:

public static IQueryable<T> ReWrite<T>(this IQueryable<T> qu)
{
  var result = new InterpolationStringReplacer<T>().Visit(qu.Expression);
  var s = (IQueryable<T>) qu.Provider.CreateQuery(result);
  return s; 
}

Pay your attention to cast qu.Provider.CreateQuery(result) that has the IQueryable method in IQueryable<T>. It is widely used for C# (look at IEnumerable<T> interface!), and it came from the need to handle all generic interfaces with one class that wants to get IQueryable/IEnumerable, and handle it using general interface methods.

We could’ve avoided that by bringing T up to a baseclass (through covariance), but it sets some limits on interface methods.

Result

Apply ReWrite to the linq expression on the top of the article:

var result = _context.Humans
                      .Select(x => $"Name: {x.Name}  Age: {x.Age}")
                      .Where(x => x != "")
                      .ReWrite()
                      .ToList();
// correct
// [Name: "Piter" Age: 19]

GitHub

Let's help QueryProvider deal with interpolated strings