3

Face registration is an important task in face recognition system . I knew how to register face by two eye-center points like this. But i don not know how to use more than two point (e.g. two eye-center, nose tip and two mouth corner) to do a face registration.

Any idea for that? Thanks in advance!

tidy
  • 4,747
  • 9
  • 49
  • 89

2 Answers2

3

If you have a lot of points, say 68..then you can perform delaunay triangulation and then perform piecewise affine warp.

If you have much fewer than 68, say 5 or 6, then you can try least square fitting of affine or perspective transform. I believe you can use the findhomography function of opencv and then use perspectivetransform function to perform this step.

Zaw Lin
  • 5,629
  • 1
  • 23
  • 41
  • Hi @zaw-lin, i have an face crop with 68 landmarks and trying to align it like a profle/frontal face. i had done delaunay triangulation .but not sure how to do piecewise affine warp. is it done with respect to a frontal face mesh ?. i see a [function](https://scikit-image.org/docs/dev/auto_examples/transform/plot_piecewise_affine.html) in scikit-image. But it takes only few points as input. kinldy help with any reference or pointers on how to do piecewise affine warp after delaunay triangulation – Bhupathy A P Jul 26 '19 at 10:37
  • Hey that was a long time ago, the technique to do it to it triangle by triangle, you do billinear interpolation for each triangle (detected triangles warp to the "canonical" standard face triangles). I did it in c++ myself so I don't know which library has the same function. Btw, this actually gives worse performance, especially with deep learning face comparisons, and it's a little troublesome as you also need to train using this preprocessing. You are probably better off just using similarity transform to align faces – Zaw Lin Aug 02 '19 at 22:59
2

For 2D alignment - to discover the affine transform that maps a set of landmark points onto another set - you are probably best starting with the classic Procrustes Analysis.

Here someone very graciously provides a converted implementation (from Matlab) into python.

Using this, here's how I can do what I think you are after...

import procrustes as pc 
import numpy as np 
import matplotlib.pyplot as plt
import matplotlib.cm as cm 
import cv2

# Open images...
target_X_img = cv2.imread('arnie1.jpg',0)
input_Y_img = cv2.imread('arnie2.jpg',0)

# Landmark points - same number and order!
# l eye, r eye, nose tip, l mouth, r mouth
X_pts = np.asarray([[61,61],[142,62],[101,104],[71,143],[140,139]])
Y_pts = np.asarray([[106,91],[147,95],[129,111],[104,130],[141,135]])

# Calculate transform via procrustes...
d,Z_pts,Tform = pc.procrustes(X_pts,Y_pts)

# Build and apply transform matrix...
# Note: for affine need 2x3 (a,b,c,d,e,f) form
R = np.eye(3)
R[0:2,0:2] = Tform['rotation']
S = np.eye(3) * Tform['scale'] 
S[2,2] = 1
t = np.eye(3)
t[0:2,2] = Tform['translation']
M = np.dot(np.dot(R,S),t.T).T
tr_Y_img = cv2.warpAffine(input_Y_img,M[0:2,:],(400,400))

# Confirm points...
aY_pts = np.hstack((Y_pts,np.array(([[1,1,1,1,1]])).T))
tr_Y_pts = np.dot(M,aY_pts.T).T

# Show result - input transformed and superimposed on target...
plt.figure() 
plt.subplot(1,3,1)
plt.imshow(target_X_img,cmap=cm.gray)
plt.plot(X_pts[:,0],X_pts[:,1],'bo',markersize=5)
plt.axis('off')
plt.subplot(1,3,2)
plt.imshow(input_Y_img,cmap=cm.gray)
plt.plot(Y_pts[:,0],Y_pts[:,1],'ro',markersize=5)
plt.axis('off')
plt.subplot(1,3,3)
plt.imshow(target_X_img,cmap=cm.gray)
plt.imshow(tr_Y_img,alpha=0.6,cmap=cm.gray)
plt.plot(X_pts[:,0],X_pts[:,1],'bo',markersize=5) 
plt.plot(Z_pts[:,0],Z_pts[:,1],'ro',markersize=5) # same as...
plt.plot(tr_Y_pts[:,0],tr_Y_pts[:,1],'gx',markersize=5)
plt.axis('off')
plt.show()

enter image description here

All this only holds true, of course, for planar/rigid and affine transforms. As soon as you start having to cater for non-affine/perspective and deformable surfaces, well, that is a entirely different topic...

Community
  • 1
  • 1
timlukins
  • 2,694
  • 21
  • 35