From what I can recall is that you create the square frames as normal but the images alpha out the areas not needed to be displayed.
You might want to have a look at how Blizzard does their frames that have circular images on.
https://www.townlong-yak.com/framexml/live
Find a frame and then locate the files likely to hold their layout ( xml ) and functionality ( lua ) from the list of files in the link above.