Building a lipstick Try-On App using Media pipe Face mesh in python
This blog post is about the project that utilizes face mesh technology to create an interactive lipstick try-on experience. The project uses Google’s media pipe face mesh solution to detect the user’s lips and apply different shades of lipstick to them in real-time. The post discusses the challenges of developing the algorithm and creating a user-friendly interface. Its ability to provide users with a unique and innovative way to experiment with different shades of lipstick. The inspiration behind this project is to provide a new and innovative way for users to experiment with different with different shades of lipstick without having to physically apply them .By using this app it an remove various problems like inconvenience , Hygiene concerns , limited selection , Uncertainty .
Background: This project Utilizes Google’s Media Pipe face mesh solution to detect and track the user’s lips. Media pipe is an open source framework for building multimodal machine learning pipelines, including face detection, facial landmark detection, and object detection.
Face mesh is a real-time face mesh solution that can be used to detect facial landmarks in real-time. The face mesh model outputs 468 landmark points on the user’s face, which can be used to track various facial features, including lips
Specific Algorithm used : To apply different shades of lipstick to the user’s lips, this project uses a combination of computer vision techniques and deep learning algorithm. First, the face mesh solution is used to detect and track the user’s lips, Then a color filter is applied to the lip region, which isolates the lip color from the rest of the image.
Overall, this algorithm combines multiple computer vision and deep learning techniques to create a realistic and interactive try-on experience
Visualization
The step-by step procedure of this project is as follows
Graphical User Interface (GUI): We will start by creating a Graphical User Interface using the Tkinter library in python. The GUI will have three frames. The first frame will display the app’s logo and a button to navigate to the second frame, where the user can select the lipstick shade they want to try-on. The second frame will have list of available lipstick shades and a user can select one. The third frame will display the live camera feed and overlay the selected the selected lipstick shade on the user’s lips
Let’s start by importing the required libraries and setting up some variables:
import tkinter as tk
from PIL import ImageTk
import cv2
# set colours
bg_colour = "#3d6466"
# create shades
shades = ["RED", "NUDE", "PINK", "PLUM", "CORAL", "BERRY", "PEACH"]We will use the Tkinter library to create the GUI, and the PIL and OpenCV libraries to work with images and camera feed. We set the background color of GUI to dark teal color , and we create a list of lipstick shades that the user can choose from
Next, we define the ‘Camera’ class, which will be the main application:
def clear_widgets(frame):
# select all frame widgets and delete them
for widget in frame.winfo_children():
widget.destroy()
class Camera:
def _init_(self, device=0, width=400, height=300):
self.device = device
self.width = width
self.height = height
self.cap = cv2.VideoCapture(self.device)
self.cap.set(cv2.CAP_PROP_FRAME_WIDTH, self.width)
self.cap.set(cv2.CAP_PROP_FRAME_HEIGHT, self.height)
def _del_(self):
self.cap.release()
def get_frame(self):
ret, frame = self.cap.read()
if not ret:
raise Exception("Failed to capture frame")
return frame
def update_canvas(self, canvas):
# Get a frame from the camera
ret, frame = self.cap.read()
if not ret:
raise Exception("Failed to capture frame")
# fliiping the camera
frame=cv2.flip(frame,1)
# Convert the frame from BGR to RGB format
frame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
# Convert the frame to a PIL ImageTk format
img = Image.fromarray(frame)
imgtk = ImageTk.PhotoImage(image=img)
# Update the canvas with the new image
canvas.imgtk = imgtk
canvas.create_image(0, 0, anchor=tk.NW, image=imgtk)
# Reschedule the update
canvas.after(5, lambda: self.update_canvas(canvas))In the ‘__init__’ method, we initialize the main application window ,set its title , and center it on the screen. We create two frames ‘Frame 1’ will create the app’s logo and a button to navigate to the second frame , and ‘Frame 2’ will contain a list of lipstick shades. We use a for loop to place a frame in main window. Finally we call the ‘load_frame1’ method to load the first frame.
The ‘load_frame1’ method displays the app’s logo and a button to navigate to the second frame.
def load_frame1():
clear_widgets(frame2)
# stack frame 1 above frame 2
frame1.tkraise()
# prevent widgets from modifying the frame
frame1.pack_propagate(False)
# create logo widget
logo_img = ImageTk.PhotoImage(file="C:/Users/zahoo/New folder/luxe5.png")
logo_widget = tk.Label(frame1, image=logo_img, bg=bg_colour)
logo_widget.image = logo_img
logo_widget.pack(pady=30)
# create label widget for instructions
tk.Label(
frame1,
text="Ready for some Lipstick?",
bg=bg_colour,
fg="white",
font=("Shanti", 14)
).pack(pady=15)
# create button widget
tk.Button(
frame1,
text="SHADES",
font=("Ubuntu", 20),
bg="#28393a",
fg="white",
cursor="hand2",
activebackground="#badee2",
activeforeground="black",
command=lambda:load_frame2()
).pack(pady=20)A part of this code is used to load and display the first frame of a GUI application. The first line of the method ‘self.clear.widgets(self.frame2)’ clears any widgets that may be present on the second frame of the GUI application. This is done to ensure that the second frame is not displayed when first frame is loaded
The next line ‘self.frame1.tkraise()’ brings the first frame to the top of the stack, making it the visible frame of the GUI application.
The third line ‘self.frame1.pack_propagate(False)’ prevents widgets from modifying the frame’s dimensions
The next few lines of the code create and display various widgets on the first frame.
The first widget created is a logo widget which displays an image. The image file is loaded using the ‘ImageTk.PhotoImage()’ method and is assigned to ‘logo_img’ variable . The ‘tk.Label()’ method is used to create a label widget and display image on GUI . The ‘logo_widget.image = logo_img’ line ensure that image is not garbage collected.
The second widget is a label widget that displays some text. This widget is created using the ‘tk.Label()’ method and is used to display the instruction message to the user.
The third widget is a button widget that allows user to navigate to the second frame of GUI application. This widget is created using the ‘tk.Button()’ and is assigned a command to call the ‘load_frame2()’ method when clicked.
The‘ load_frame2()’ method is responsible for displaying a list of shades of lipstick in the user interface. It clears any widgets in the ‘frame1’ , sets ‘frame2’ to be the topmost frame, and creates a logo widget to display an image. It then iterates over a list of shade names and create a button widget for each shade. Clicking on the shade button triggers the ‘load_frame3()’ method. Finally it creates a ‘BACK’ button that, when clicked, triggers the ‘load_frame1()’ method to return to the previous frame.
Widgets created in this method include:
- ‘logo_widget’ : A ‘tk.Label1’ widget to display an image
- Shade button widgets : Multiple ‘tk.button’ widgets each displaying different shade names
- ‘BACK’ button : A ‘tk.button” widget to return to previous frame
def load_frame2():
clear_widgets(frame1)
# stack frame 2 above frame 1
frame2.tkraise()
# create logo widget
logo_img = ImageTk.PhotoImage(file="C:/Users/zahoo/New folder/luxe5.png")
logo_widget = tk.Label(frame2, image=logo_img, bg=bg_colour)
logo_widget.image = logo_img
logo_widget.pack(pady=20)
for i in shades:
# create shade button widget
tk.Button(
frame2,
text=i,
font=("Ubuntu", 12),
bg="#28393a",
fg="white",
cursor="hand2",
activebackground="#badee2",
activeforeground="black",
width=10, height=1,
command=lambda:load_frame3()
).pack(pady=5)
tk.Button(
frame2,
text="BACK",
font=("Ubuntu", 9),
bg="#28393a",
fg="white",
cursor="hand2",
activebackground="#badee2",
activeforeground="black",
command=lambda:load_frame1()
).pack(pady=6)
The‘ load_frame3() ’function initializes the camera by creating a Video Capture Object. The function enters the loop where it captures each frame of camera feed by calling the read method on the video capture object. When the user presses ‘q’ the loop breaks and camera is released using the ‘release’ method of Video Capture Object .Finally all openCV windows are closed using the ‘destroyAllwindows’ function
def load_frame3():
clear_widgets(frame2)
clear_widgets(frame3)
# stack frame 2 above frame 1
frame3.tkraise()
# create logo widget
logo_img = ImageTk.PhotoImage(file="C:/Users/zahoo/New folder/luxe5.png")
logo_widget = tk.Label(frame3, image=logo_img, bg=bg_colour)
logo_widget.image = logo_img
logo_widget.pack(pady=5)
# Create a quit button
tk.Button(
frame3,
text="QUIT",
font=("Ubuntu", 9),
bg="#28393a",
fg="white",
cursor="hand2",
activebackground="#badee2",
activeforeground="black",
width=15, height=2,border=2,
command=lambda:load_frame2()
).pack(pady=6,side="bottom")
# create canvas widget
canvas = tk.Canvas(frame3, width=400, height=300, bg='black')
canvas.pack()
# create camera object and start updating the canvas
camera = Camera(device=0, width=400, height=300)
camera.update_canvas(canvas)
# initiallize app with basic settings
root = tk.Tk()
root.title("Luxe Lips")
# root.eval("tk::PlaceWindow . center")
root.configure(background=bg_colour)
# place app in the center of the screen (alternative approach to root.eval())
x = root.winfo_screenwidth() // 2
y = int(root.winfo_screenheight() * 0.1)
root.geometry('500x600+' + str(x) + '+' + str(y))
# create a frame widgets
frame1 = tk.Frame(root, width=500, height=600, bg=bg_colour)
frame2 = tk.Frame(root, bg=bg_colour)
frame3 = tk.Frame(root, bg=bg_colour)
# place frame widgets in window
for frame in (frame1, frame2, frame3):
frame.grid(row=0, column=0, sticky="nesw")
# load the first frame
load_frame1()
# run app
root.mainloop()
After creating GUI next step is to add face mesh solution of media pipe in the code of GUI. There are three frames we created in the GUI of this project, the face mesh solution will be added in the frame 3 which is the camera. When camera means frame 3 is open we wil add facemesh solution of media pipe. When the user comes in front of the camera the face mesh will be shown in the form of facial landmarks. In this project we are interested only in the lips landmark in which a user can try different shades of lipstick.
When a user click on shade of lipstick which wants to be try , frame 3 of camera open and user experience the lipstick shade .
Final code :
import cv2
import tkinter as tk
from PIL import ImageTk, Image
import numpy as np
import mediapipe as mp
# set colours
bg_colour = "#3d6466"
# create shades
shades = ["RED", "NUDE", "PINK", "PLUM", "CORAL", "BERRY", "PEACH"]
def clear_widgets(frame):
# select all frame widgets and delete them
for widget in frame.winfo_children():
widget.destroy()
mp_drawing = mp.solutions.drawing_utils
mp_face_mesh = mp.solutions.face_mesh
class Camera:
def _init_(self, device=0, width=400, height=300):
self.device = device
self.width = width
self.height = height
self.cap = cv2.VideoCapture(self.device)
self.cap.set(cv2.CAP_PROP_FRAME_WIDTH, self.width)
self.cap.set(cv2.CAP_PROP_FRAME_HEIGHT, self.height)
self.mp_drawing = mp_drawing
self.mp_face_mesh = mp_face_mesh
self.lip_landmarks = [61, 62, 63, 64, 65, 66, 67, 0, 146, 91, 181, 84, 17, 314, 405, 321, 375, 291, 61]
def _del_(self):
self.cap.release()
def get_frame(self):
ret, frame = self.cap.read()
if not ret:
raise Exception("Failed to capture frame")
return frame
def update_canvas(self, canvas, shade):
# Get a frame from the camera
ret, frame = self.cap.read()
if not ret:
raise Exception("Failed to capture frame")
# flipping the camera
frame=cv2.flip(frame,1)
# Convert the frame from BGR to RGB format
frame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
# Detect the face mesh
with mp_face_mesh.FaceMesh(
max_num_faces=1,
min_detection_confidence=0.5,
min_tracking_confidence=0.5) as face_mesh:
# Flip the image horizontally for a later selfie-view display
image = cv2.flip(frame, 1)
# Convert the image to RGB
image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)
# To improve performance, optionally mark the image as not writeable to
# pass by reference.
image.flags.writeable = False
results = face_mesh.process(image)
# Draw the face mesh annotations on the image.
image.flags.writeable = True
annotated_image = cv2.cvtColor(image, cv2.COLOR_RGB2BGR)
if results.multi_face_landmarks:
for face_landmarks in results.multi_face_landmarks:
# Extract the lip landmarks
lip_pts = np.array([(landmark.x, landmark.y) for landmark in face_landmarks.landmark])[self.lip_landmarks]
# Scale the lip points to the size of the frame
lip_pts[:, 0] *= frame.shape[1]
lip_pts[:, 1] *= frame.shape[0]
# Create a mask for the lips
lip_mask = np.zeros(frame.shape[:2], dtype=np.uint8)
cv2.drawContours(lip_mask, [lip_pts.astype(np.int32)], -1, 255, -1)
for i in shades:
# Apply the lipstick shade to the lips
if i == "RED":
lipstick = (0, 0, 255)
elif i == "NUDE":
lipstick = (255, 192, 203)
elif i == "PINK":
lipstick = (147, 20, 255)
elif i == "PLUM":
lipstick = (221, 160, 221)
elif i == "CORAL":
lipstick = (255, 127, 80)
elif i == "BERRY":
lipstick = (128, 0, 128)
elif i == "PEACH":
lipstick = (255, 218, 185)
frame[lip_mask != 0] = lipstick
# Convert the frame to a PIL ImageTk format
img = Image.fromarray(frame)
imgtk = ImageTk.PhotoImage(image=img)
# Update the canvas with the new image
canvas.imgtk = imgtk
canvas.create_image(0, 0, anchor=tk.NW, image=imgtk)
# Reschedule the update
canvas.after(5, lambda: self.update_canvas(canvas, shade))
def load_frame1():
clear_widgets(frame2)
# stack frame 1 above frame 2
frame1.tkraise()
# prevent widgets from modifying the frame
frame1.pack_propagate(False)
# create logo widget
logo_img = ImageTk.PhotoImage(file="C:/Users/zahoo/New folder/luxe5.png")
logo_widget = tk.Label(frame1, image=logo_img, bg=bg_colour)
logo_widget.image = logo_img
logo_widget.pack(pady=30)
# create label widget for instructions
tk.Label(
frame1,
text="Ready for some Lipstick?",
bg=bg_colour,
fg="white",
font=("Shanti", 14)
).pack(pady=15)
# create button widget
tk.Button(
frame1,
text="SHADES",
font=("Ubuntu", 20),
bg="#28393a",
fg="white",
cursor="hand2",
activebackground="#badee2",
activeforeground="black",
command=lambda:load_frame2()
).pack(pady=20)
def load_frame2():
clear_widgets(frame1)
# stack frame 2 above frame 1
frame2.tkraise()
# create logo widget
logo_img = ImageTk.PhotoImage(file="C:/Users/zahoo/New folder/luxe5.png")
logo_widget = tk.Label(frame2, image=logo_img, bg=bg_colour)
logo_widget.image = logo_img
logo_widget.pack(pady=20)
for i in shades:
# create shade button widget
tk.Button(
frame2,
text=i,
font=("Ubuntu", 12),
bg="#28393a",
fg="white",
cursor="hand2",
activebackground="#badee2",
activeforeground="black",
width=10, height=1,
command=lambda:load_frame3(i)
).pack(pady=5)
tk.Button(
frame2,
text="BACK",
font=("Ubuntu", 9),
bg="#28393a",
fg="white",
cursor="hand2",
activebackground="#badee2",
activeforeground="black",
command=lambda:load_frame1()
).pack(pady=6)
def load_frame3(shades):
clear_widgets(frame2)
clear_widgets(frame3)
# stack frame 2 above frame 1
frame3.tkraise()
# create logo widget
logo_img = ImageTk.PhotoImage(file="C:/Users/zahoo/New folder/luxe5.png")
logo_widget = tk.Label(frame3, image=logo_img, bg=bg_colour)
logo_widget.image = logo_img
logo_widget.pack(pady=5)
# Create a quit button
tk.Button(
frame3,
text="QUIT",
font=("Ubuntu", 9),
bg="#28393a",
fg="white",
cursor="hand2",
activebackground="#badee2",
activeforeground="black",
width=15, height=2,border=2,
command=lambda:load_frame2()
).pack(pady=6,side="bottom")
# create canvas widget
canvas = tk.Canvas(frame3, width=400, height=300, bg='black')
canvas.pack()
# create camera object and start updating the canvas
camera = Camera()
camera.update_canvas(canvas, shades)
# initiallize app with basic settings
root = tk.Tk()
root.title("Luxe Lips")
# root.eval("tk::PlaceWindow . center")
root.configure(background=bg_colour)
# place app in the center of the screen (alternative approach to root.eval())
x = root.winfo_screenwidth() // 2
y = int(root.winfo_screenheight() * 0.1)
root.geometry('500x600+' + str(x) + '+' + str(y))
# create a frame widgets
frame1 = tk.Frame(root, width=500, height=600, bg=bg_colour)
frame2 = tk.Frame(root, bg=bg_colour)
frame3 = tk.Frame(root, bg=bg_colour)
# place frame widgets in window
for frame in (frame1, frame2, frame3):
frame.grid(row=0, column=0, sticky="nesw")
# load the first frame
load_frame1()
# run app
root.mainloop()