Good fences don't make good neighbors in the world of Game of Thrones...

Great explanation of why the free folk want to be on the other side of the Wall:

But the Long Night wasn't just a tale to scare children. It was the reason the Night's Watch was established in the first place, and it was the reason the Wall — 500 miles of ice wall, 700 feet high — was built some 8,000 years ago. Some of those who know that the White Walkers did exist believe they've been gone since then. But as a wildling trying to flee South explains, "They wasn't gone. They were sleeping, and they ain't sleeping no more." Some of those who know the White Walkers have woken up again try to keep them at bay — Craster sacrificed some 99 of his male children to the Others, so that he and his wives/daughters might be left in peace. The late Lord Commander Mormont knew this as well, and agreed to look the other way, telling Jon Snow, "Wildlings serve crueler gods than you or I."

Most wildlings, however, refuse to serve anyone. And while we're at it, they prefer not to be called wildlings — they call themselves the free folk. In turn, they call those in Westeros "kneelers," because they look down on those who would serve a king. Their own, the King Beyond the Wall, is not a king at all, but the closest thing to an elected official you might find in this world. "We don't go serving some shit king who is only king because of who his father was," Ygritte tells Jon Snow. "We chose Mance Rayder to lead us." And as Mance tells Jon Snow — it seems everyone's having to tell Jon Snow something! — it's not easy leading a massive group of people who define themselves by their very freedom from leaders. "You know what it takes to unite 90 clans, half of whom want to massacre the other half over one insult or another?" he asks. "Do you know I got moon worshippers, cannibals, and giants to march together in the same army? I told them we were all going to die if we don't get South, because that's the truth."

These various wildling tribes — the Thenns, the Hornfoots, the Ice River Clans, the Cave People, and more — know that the White Walkers have woken up. Some joined Mance's army; others like Osha and her former traveling companions tried to flee South on their own. "Piss on Mance Rayder and piss on the North," one of them tells her. "We're going as far south as south goes. There aren't no White Walkers down in Dorne."

If it weren't for the Wall, the White Walkers could migrate that far south, too, as the Wall is Westeros's protection against them. But it's also what prevents a mass exodus of wildlings fleeing the far North, stemming the tide of would-be illegal immigrants trapped beyond the Wall who want to find safer ground. As Ygritte complains to Jon, the land should be available to all of them. "Your lot just came along and put up a big wall and said it was yours!"  They argue about who has been invading whom — the people in Westeros look at the wildlings as savages and raiders, but the free folk have more of a "this land is our land" approach. "Why are you fighting us?" she asks pointedly. Tyrion also believes the main difference "between us and the wildlings is that when the Wall went up, our ancestors happened to live on the right side of it."

Yet over the years, the White Walkers faded from memory and the idea of them was treated as mere superstition. Even when the Night's Watch asked the kings of  Westeros for some serious backup, their requests fell on deaf ears — the assumption was that the Night's Watch was there to protect them against wildlings, not a serious threat. (If they came over the Wall to raid a village and steal supplies, it was a localized affair, and any attempts to breach the Wall en masse had failed). The rangers continued to patrol the wilderness — tracking and sometimes fighting and killing the wildlings, which of course only antagonized them further. (The orphaned wildling boys pelted Jon Snow, for instance, since they had lost their fathers to crows).

Those patrols North of the Wall were perceived by wildlings as an invasion, and those South of the Wall thought of it as a joke. "Is that what we are to you, an army of jesters in black?" Benjen Stark defensively asks Tyrion. The Starks may have kept the tradition of noble families volunteering for service, but the order was no longer perceived as an elite force by the rest of Westeros, and their numbers dwindled, forcing them to find recruits among runaways and criminals. "Ah, rapers," Tyrion notes. "They were given a choice, no doubt. Castration, or the Wall. Most choose the knife."  Those who choose the Wall don't always make it there — Yoren's last group of recruits from the dungeons of King's Landing were waylaid en route. Still more of the Night's Watch died battling both each other (during the mutiny at Craster's Keep) and the White Walkers (during ranging trips and at the Fist of the First Men). So even though Jon Snow lied to Mance Rayder, claiming that a thousand men guarded Castle Black, that was no longer true. And only 3 of the 19 castles — Castle Black, the Shadow Tower, and Eastwatch-by-the-Sea — have regiments. Hardly enough to fight both Mance's army and an army of the dead. Instead of fighting each other, shouldn't both the Night's Watch and Mance's army of wildlings take on their common enemy? Years of antagonism and resentment, however, have led to this moment. And if a hundred thousand wildlings want a way around the Wall into Westeros to avoid the White Walkers, the gate at Castle Black is their best option.

All major characters in Night's Watch and Free Folk:

http://pandawhale.com/post/45196/image-of-current-known-nights-watch-members-in-preparation-of-game-of-thrones-s4e9