How to shrink/expand facial features using Opencv?

Question

I'm trying write an application make parts of face image bigger or smaller with opencv and dlib. I detect facial landmarks using shape_predictor_68_face_landmarks.dat. In the following function, the tmp variable is supposed to be transformed in such a way that scale nose or left eye on image.

def visualize_facial_landmarks(image, shape, colors=None, alpha=0.75):
# create two copies of the input image -- one for the
# overlay and one for the final output image
overlay = image.copy()
output = image.copy()

# if the colors list is None, initialize it with a unique
# color for each facial landmark region
if colors is None:
    colors = [(19, 199, 109), (79, 76, 240), (230, 159, 23),
              (168, 100, 168), (158, 163, 32),
              (163, 38, 32), (180, 42, 220)]

# loop over the facial landmark regions individually
for (i, name) in enumerate(FACIAL_LANDMARKS_INDEXES.keys()):
    # grab the (x, y)-coordinates associated with the
    # face landmark
    (j, k) = FACIAL_LANDMARKS_INDEXES[name]
    pts = shape[j:k]
 
    facial_features_cordinates[name] = pts
    if name != "Jaw" and name == "Left_Eye" or name == "Nose":
        minX = min(pts[:,0])
        maxX = max(pts[:,0])
        minY = min(pts[:,1])
        maxY = max(pts[:,1]) 

        rect = []
        rect.append([minX, minY])
        rect.append([minX, maxY])
        rect.append([maxX, minY])
        rect.append([maxX, maxY])
        rect = np.array(rect)

        hull = cv2.convexHull(rect)
        # print(hull)
        # output = cv2.resize(overlay, dsize)
        # print(overlay[minX:maxX,minY:maxX,:])
        tmp = overlay[minY:maxY, minX:maxX, :]
        print(tmp.shape)
        s = 2

        Affine_Mat_w = [s, 0, tmp.shape[0]/2.0 - s*tmp.shape[0]/2.0]
        Affine_Mat_h = [0, s, tmp.shape[1]/2.0 - s*tmp.shape[1]/2.0]


        M = np.c_[ Affine_Mat_w, Affine_Mat_h].T 

        tmp = cv2.warpAffine(tmp, M, (tmp.shape[1], tmp.shape[0]))
        overlay[minY:maxY, minX:maxX, :] = tmp

return overlay

As an example the following image attached:

B200011011 · Accepted Answer · 2021-11-10T06:09:52.920

Update #1

Applying pinch and bulge distortion along facial landmarks in small amounts around eyes and nose could probably provide decent results without moving into another method. Though there is a chance it will also noticeably distort eyeglasses if it affects a larger area. These should help,

I am not sure how to do this in opencv alone without face looking unnatural. Below is a general explanation based on my own exploration. Feel free to correct me if anything if I made any mistake.

3D Mesh

One way I think current face beautification methods such as those on Android cameras work is to align a 3d face mesh or a whole head model on top of original face.

It extracts face texture using using facial landmarks and aligns them with corresponding 3d mesh with texture applied to it. This way the 3d mesh can be adjusted and texture will follow face geometry. There are probably additional steps such as passing the result to another network, post-processing involved to make it look more natural.

Mediapipe Face Mesh will probably be helpful also as it provides dense 3d face landmarks with 3D face models, UV visualization, coordinates. This is a discussion for UV unwrapping of face in mediapipe.