Julia scope of macro variables and eval

Question

I wanted make a macro that creates some code for me. E.g.

I have a vector x = [9,8,7] and I want to use a macro to generate this piece of code vcat(x[1], x[2], x[3]) and run it. And I want it to work for arbitrary length vectors.

I have made the macro as below

macro some_macro(a)
  quote
    astr = $(string(a))
    s = mapreduce(aa -> string(astr,"[",aa,"],"), string, 1:length($(a)))
    eval(parse(string("vcat(", s[1:(end-1)],")")))
  end
end

x = [7,8,9]
@some_macro x

The above works. But when I try to wrap it inside a function

function some_fn(y)
  @some_macro y
end

some_fn([4,5,6])

It doesn't work and gives error

UndefVarError: y not defined

and it highlights the below as the culprit

s = mapreduce(aa -> string(astr,"[",aa,"],"), string, 1:length($(a)))

Edit See julia: efficient ways to vcat n arrays

for advanced example why I want to do instead of using the splat operator

Possible duplicate of [Julia v0.6 macro inside function](https://stackoverflow.com/questions/45400875/julia-v0-6-macro-inside-function) — Gnimuc, Sep 27 '17 at 14:21
you need to `esc`ape `a`: `astr = string($(esc(a)))` and make the same change to the second line. — Gnimuc, Sep 27 '17 at 14:22
have you checked out `@generated` functions? it seems like that's what you really need here, not macro. — Gnimuc, Sep 27 '17 at 14:28

mbauman · Accepted Answer · 2017-09-29T02:00:31.137

You don't really need macros or generated functions for this. Just use vcat(x...). The three dots are the "splat" operator — it unpacks all the elements of x and passes each as a separate argument to vcat.

Edit: to more directly answer the question as asked: this cannot be done in a macro. Macros are expanded at parse time, but this transformation requires you to know the length of the array. At global scope and in simple tests it may appear that it's working, but it's only working because the argument is defined at parse time. In a function or in any real use-cases, however, that's not the case. Using eval inside a macro is a major red flag and really shouldn't be done.

Here's a demo. You can create a macro that vcats three arguments safely and easily. Note that you should not construct strings of "code" at all here, you can just construct an array of expressions with the :( ) expression quoting syntax:

julia> macro vcat_three(x)
           args = [:($(esc(x))[$i]) for i in 1:3]
           return :(vcat($(args...)))
       end
@vcat_three (macro with 1 method)

julia> @macroexpand @vcat_three y
:((Main.vcat)(y[1], y[2], y[3]))

julia> f(z) = @vcat_three z
       f([[1 2], [3 4], [5 6], [7 8]])
3×2 Array{Int64,2}:
 1  2
 3  4
 5  6

So that works just fine; we esc(x) to get hygiene right and splat the array of expressions directly into the vcat call to generate that argument list at parse time. It's efficient and fast. But now let's try to extend it to support length(x) arguments. Should be simple enough. We'll just need to change 1:3 to 1:n, where n is the length of the array.

julia> macro vcat_n(x)
           args = [:($(esc(x))[$i]) for i in 1:length(x)]
           return :(vcat($(args...)))
       end
@vcat_n (macro with 1 method)

julia> @macroexpand @vcat_n y
ERROR: LoadError: MethodError: no method matching length(::Symbol)

But that doesn't work — x is just a symbol to the macro, and of course length(::Symbol) doesn't mean what we want. It turns out that there's absolutely nothing you can put there that works, simply because Julia cannot know how large x is at compile time.

Your attempt is failing because your macro returns an expression that constructs and evals a string at run-time, and eval does not work in local scopes. Even if that could work, it'd be devastatingly slow… much slower than splatting.

If you want to do this with a more complicated expression, you can splat a generator: vcat((elt[:foo] for elt in x)...).

Gnimuc · Answer 2 · 2017-09-28T00:03:30.300

FWIW, here is the @generated version I mentioned in the comment:

@generated function vcat_something(x, ::Type{Val{N}}) where N
    ex = Expr(:call, vcat)
    for i = 1:N
        push!(ex.args, :(x[$i]))
    end
    ex
end

julia> vcat_something(x, Val{length(x)})
5-element Array{Float64,1}:
 0.670889 
 0.600377 
 0.218401 
 0.0171423
 0.0409389

You could also remove @generated prefix to see what Expr it returns:

julia> vcat_something(x, Val{length(x)})
:((vcat)(x[1], x[2], x[3], x[4], x[5]))

Take a look at the benchmark results below:

julia> using BenchmarkTools

julia> x = rand(100)

julia> @btime some_fn($x)
  190.693 ms (11940 allocations: 5.98 MiB)

julia> @btime vcat_something($x, Val{length(x)})
  960.385 ns (101 allocations: 2.44 KiB)

The huge performance gap is mainly due to the fact that @generated function is firstly executed and executed only once at compile time(after the type inference stage) for each N that you passed to it. When calling it with a vector x having the same length N, it won't run the for-loop, instead, it'll directly run the specialized compiled code/Expr:

julia> x = rand(77);  # x with a different length

julia> @time some_fn(x);
  0.150887 seconds (7.36 k allocations: 2.811 MiB)

julia> @time some_fn(x); 
  0.149494 seconds (7.36 k allocations: 2.811 MiB)

julia> @time vcat_something(x, Val{length(x)});
  0.061618 seconds (6.25 k allocations: 359.003 KiB)

julia> @time vcat_something(x, Val{length(x)});
  0.000023 seconds (82 allocations: 2.078 KiB)

Note that, we need to pass the length of x to it ala a value type(Val), since Julia can't get that information(unlike NTuple, Vector only has one type parameter) at compile time.

EDIT: see Matt's answer for the right and simplest way to solve the problem, I gonna leave the post here since it's relevant and might be helpful when dealing with splatting penalty.

Julia scope of macro variables and eval

2 Answers2