A/B/O fic - named after the Alpha/Beta/Omega divisions of canine pack hierarchies, is a sub-genre of fic that became popular in the fandom from mid-2010 onwards. The genre overlays canine-type characteristics on people, including pack hierarchies and the idea that receptive sex partners may go into heat. Males have a penis that forms a knot when erect, as is found in dogs and wolves. The act of sex when the knot swells is referred to as "knotting".
There is no single accept set of 'rules' for stories written in the genre, but they usually includes strong themes of dominance and submission, and possessiveness. Some stories will build alternate societal and cultural traditions, while others may be pretty similar to real life.
While there is at least one early fic (from Star Trek: Next Generation - see Fanlore entry below) fandom which utilised a similar feature, Supernatural fandom is generally credited with having started the trope. It originated in one of the fandom's kink memes with the following prompt:
AU - Their world is just like ours...except...in their world there are two types of men. One is the alpha male, the other is the beta male. Alpha males are like any ordinary guy with the exception of their cocks, they work just like dog cocks (the knot, tons of cum etc) The beta male, is just an ordinary guy without the special cock.I'd like to see Alpha male Jared, and Beta male Jensen. Jensen is a snotty prude (think Lady from lady and the tramp) he may be a beta male but he's not just going to let anybody take a go at his sweet little ass...until he meets Jared...then prudey little Jensen turns cock slut for Jared. Bonus points for J2 being OTP, Jensen was a virgin before Jared, and now that they met each other, it's for life. Completely up to you if mating happens just anywhere like in the middle of the sidewalk, in a park etc or on a more private level. source.
Early fic, often classified in a genre called "animal traits" arose in kink fic communities and so had strong themes of extreme Dominance/submission, possessiveness and humiliation. It was influenced by writing in both bestiality and werewolf fic. As the A/B/O tropes became overlaid on all genres of mainstream fanfic, beyond the kink space, these themes became more mild.
- The nonnies made them do it! The fannish history. SPN. A/B/O by netweight
- Omegaverse explained by MosImagination
- Alpha/Beta/Omega entry on Fanlore
- Tying the knot - an analysis of the relationship between D/S, bestiality and knotting fic
- Take the knot - fic comm on LJ
- Knotting entry on Fanlore
- There is a chapter on knotting in Fic: Why Fanfiction Is Taking Over the World and it is also mentioned in the chapter on Supernatural fic.