The quick answer is that an occupied outpost is vacated (neutralized) when its health is reduced to 0 and at this point the health is reset to its max value.
The longer answer:
Outposts have a "health" value. This health value is reduced every time damage is done to the outpost.
If the outpost is vacant (not occupied) then reducing that health value to 0 will occupy the outpost. When occupied the health is reset to its max value.
If the outpost is occupied then reducing the health value to 0 will vacate the outpost (remove its ownership) and reset it's health to its max value.
Doing more damage to an outpost than it has in health will not affect its health in its new state (e.g. doing 5 damage to an occupied outpost with 3 health and 3 max health will result in it being vacant and on 3 health).
This is the primary method of occupying outposts -- e.g. reducing the health of a vacant outpost to 0. If the outpost is occupied, you must vacate it first. This must be done in two separate damaging actions (e.g. 6 damage to a 3-health occupied outpost will not occupy it)
It's also possible to occupy outposts with "occupy", "capture" and "swap" abilities:
"Occupy" -- only works on vacant outposts, doesn't do any damage and resets health to max
"Capture" -- works on vacant or occupied outposts, resets health to max but reduces durability to the next durability state (e.g. healthy becomes distressed, distressed becomes critical, critical is destroyed)
"Swap" -- only works on occupied outposts, swaps their ownership to the other player
Some additional related concepts:
Durability -- the durability value of an outpost is reduced every time damage is done by the total damage received (e.g. 6 damage to a 3 health outpost will reduce durability by 6). When the outpost reaches 0 durability it is destroyed and cannot be occupied or used (it can be resurrected with resurrect abilities).
Outpost burnout -- most maps feature "Outpost burnout", typically 2 damage per destroyed outpost to each player at the start of their turn
Durability State -- Each outpost has a threshold for "healthy", "distressed", "critical" and "destroyed" (always 0). Some outpost abilities interface with these states (e.g. modified to have more damage) and some abilities use them (e.g. "... to all distressed outposts").
Outpost cooldowns -- outpost abilities have cooldowns (normally 1). These are reset when the occupation state changes (e.g. when occupied through damage, occupied by an ability or captured). This means you can't normally use an outpost on the turn you occupy it. The typical exception is "swap": because the occupation state doesn't change (it just changes owner) the cooldown remains as is, meaning you can "swap" an outpost and use it on the same turn if you have sufficient action points.