You can listen for GROUP_ROSTER_UPDATE events. Keep track of the group state in a variable outside of your event handler. When the group changes, figure out if your actual group state changed (the event fires for all kinds of group related reasons, not just joining/leaving one) and if it did, make the appropriate changes in your addon.
Code:
local inGroup
local frame = CreateFrame("Frame")
frame:RegisterEvent("GROUP_ROSTER_UPDATE")
frame:SetScript("OnEvent", function(self, event)
local inGroupNow = IsInGroup()
if inGroupNow == inGroup then return end -- nothing changed
inGroup = inGroupNow -- store the new state
-- make your changes here
end)
You can also get more detailed about tracking the group type/size (eg. to use different layouts in a raid group vs a battleground vs a 5-man, etc.). See
Grid's code for an example.